Hire AI & ML engineers.
LLM experts, production-ready.

Expected answer

Architecture: 1) Document ingestion pipeline: chunk documents (500-1000 tokens with overlap), generate embeddings (OpenAI ada-002 or Cohere), store in vector DB (pgvector for simplicity, Pinecone for scale). 2) Query pipeline: embed user query, retrieve top-k similar chunks, construct prompt with context + chat history, call LLM with system prompt defining tone/boundaries. 3) Guardrails: topic filtering, hallucination detection (check claims against retrieved docs), escalation to human for low-confidence responses. 4) Evaluation: use RAGAS framework (faithfulness, relevance, context recall). 5) Iteration: A/B test chunk sizes, embedding models, retrieval strategies, and prompt templates.

Expected answer

Pre-deployment: offline evaluation with held-out test set, metrics specific to the task (precision/recall for classification, RMSE for regression, BLEU/ROUGE for text). Post-deployment: track prediction distribution drift (PSI, KL divergence), feature drift, latency percentiles, error rates. Shadow deployment first, then canary rollout with automatic rollback. Alerting on: accuracy degradation, input data anomalies, serving errors. Tools: Evidently for drift detection, Prometheus for metrics, custom dashboards for business KPIs tied to model performance.

Expected answer

RAG adds knowledge at inference time by retrieving relevant documents — best for factual QA over changing data, no retraining needed. Fine-tuning modifies model weights on your data — best for teaching the model new behaviors, styles, or domain-specific reasoning. Use RAG when: data changes frequently, you need citations, or you want to avoid training costs. Use fine-tuning when: you need consistent style/format, the task requires reasoning patterns not in base model, or you need to reduce prompt size. Often combine both: fine-tune for behavior, RAG for knowledge.

Expected answer

Strategies: 1) Prompt caching (identical prompts return cached responses), 2) Semantic caching (similar queries hit cache), 3) Model routing (use smaller/cheaper models for simple queries, large models only for complex ones), 4) Prompt compression (remove redundant context), 5) Batch processing for non-real-time tasks, 6) Fine-tune smaller models on your specific task to replace expensive API calls, 7) Local open-source models (Llama, Mistral) for non-critical tasks. Monitor cost per query and quality metrics to find the optimal trade-off.

Expected answer

Architecture: Stream processing with Kafka → Feature computation (real-time: Flink/Spark Streaming for aggregates like "transactions in last 5 min"; batch: Spark for historical features) → Feature store (Feast) for consistent features between training and serving → Model serving: ensemble of gradient boosting (XGBoost for speed) + neural network (for complex patterns) behind a FastAPI service with Redis caching → Decision engine with configurable thresholds → Human review queue for borderline cases. Training: retrain weekly on rolling window, track performance by segment (merchant type, transaction amount). Key: sub-100ms latency requirement means model must be lightweight and pre-loaded in memory.