The problem

Employers in Zambia must follow the Employment Code Act of 2019, pay NAPSA and NHIMA contributions, and apply their own HR policies, and most small firms have no HR lawyer. Questions about notice, leave, gratuity and contributions come in every week, and a wrong answer means a dispute or a penalty.

This is the same production shape the source course sets for regulated-domain RAG. Hybrid retrieval with reranking, answers that cite the section, role-based access so an employee never sees payroll, a 200-question golden set labelled with an HR practitioner, a 50-prompt red team, prompt caching to keep cost per query down, a gateway with failover, tracing, and a canary release you can roll back.

Architecture

Employment Code, NAPSA/NHIMA guides, company HR policy
      |
      v
parse -> chunks + section ids + role labels (employee | hr | director)
      |
      v
dense index + BM25
      |
question + user role
      |
      v
role filter -> hybrid retrieve -> RRF -> rerank top-20 -> top-5
      |
      v
synthesise (cached prefix: contract + policies)  via gateway (primary, fallback)
      |
      v
input/output guards + PII scrub (NRC, phone, account) + citation check
      |
      v
traces -> cost in ZMW, latency, faithfulness -> dashboard
      |
      v
shadow -> canary 5% -> 50% -> 100%, rollback on gate failure

What you hand in

A golden set of 200 questions with answers and section citations, reviewed by an HR practitioner
A red-team suite of 50 prompts: payroll exfiltration, NRC and phone numbers, off-domain legal advice, injection in uploaded policies
Role-based access enforced in retrieval, tested with a 20-question cross-role probe
A cost and latency report: ZMW per query, cache hit rate, p95 latency
A dashboard for faithfulness and retrieval quality, and a written rollout record with a rollback decision

How it is marked

Faithfulness and answer relevance

Scores on the 200-question golden set

Citation correctness

Share of answers with verifiable section anchors

Guardrail coverage

Red-team pass rate and cross-role leak probe

Cost and latency engineering

Cache hit rate, p95 latency, ZMW per query

Monitoring and rollout

Live quality dashboard and a documented canary with rollback

Weights out of 100

How the industry builds it

The production pattern this capstone follows, as teams build it for regulated domains elsewhere. Read it for the tools and the trade-offs; your version runs on Zambian documents.

Build it

Ingestion. Parse your corpus (1000-10000 documents for a serious build) with Unstructured or docling. For scanned / visual-heavy pages, route through ColPali. Produce chunks with summaries, role-labels, jurisdiction tags.
Index. Dense embeddings (Voyage-3 or Nomic-embed-v2) into pgvector + pgvectorscale. BM25 side-index via Tantivy. Role and jurisdiction filters as payload.
Hybrid retrieve. Filter by role+jurisdiction first; then parallel dense + BM25; merge with reciprocal rank fusion; top-20 to reranker; top-5 to synth.
Synthesize with prompt caching. System prompt + static policies in cache header; reranked context as cache extension; user question as uncached suffix. Target 60-80% cache hit rate in steady state.
Guardrails. Llama Guard 4 on input; NeMo Guardrails rails block off-domain questions or policy-forbidden topics; Presidio scrubs accidental PII in the output; citation enforcement post-filter.
Golden set. 200 Q/A pairs labeled by a domain expert with (answer, citations). Score agent on exact-citation match, answer correctness, faithfulness (RAGAS).
Red team. 50 adversarial prompts: jailbreaks (PAIR, TAP), PII exfiltration attempts, off-domain, cross-jurisdiction leaks. Score with pass/fail and severity.
Drift dashboard. Arize Phoenix tracks retrieval quality (nDCG, citation faithfulness) weekly. Alert on 5% drop.
Cost report. Langfuse: prompt-caching hit rate, tokens per query, $/query breakdown by stage.

Exercises

Build a second corpus slice under a different jurisdiction (e.g., HIPAA alongside GDPR). Demonstrate role+jurisdiction filtering preventing cross-leak on a 20-question cross-jurisdiction probe.
Measure prompt-cache hit rate over a week of production traffic. Identify which queries break the cache prefix. Restructure.
Add multi-turn memory with a 10k-token summary buffer. Measure whether faithfulness drops as the conversation grows.
Swap Claude Sonnet 4.7 for Llama 3.3 70B self-hosted. Measure $/query and faithfulness delta.
Add an "unsure" mode: if top reranked scores are below a threshold, the agent says "I do not have confident citations" instead of answering. Measure false-confidence reduction.

Key Terms

Term	What people say	What it actually means
Prompt caching	"Cached system + context"	Claude/OpenAI feature: cached prefix tokens discounted 60-90% on hit
RAGAS	"RAG evaluator"	Automated scoring of faithfulness, answer relevance, context precision
Golden set	"Labeled eval"	200+ expert-labeled Q/A with citations; the ground truth
Jurisdiction tag	"Compliance label"	GDPR/HIPAA/SOC2 scope attached to chunks; enforced by retrieval filter
Citation faithfulness	"Grounded answer rate"	Fraction of claims backed by retrievable source spans
Drift	"Retrieval quality decay"	Weekly change in nDCG or citation score; alert threshold 5%
Red team	"Adversarial eval"	Pre-release jailbreak, PII extraction, off-domain probes

Problem

Regulated-domain RAG (legal contracts, clinical trial protocols, insurance policies) is the most-shipped production shape of 2026 because the ROI is obvious and the stakes are concrete. Harvey (Allen & Overy) built it for legal. Mendable ships the developer-docs flavor. Glean covers enterprise search. The pattern is: ingest high-fidelity, retrieve hybrid with rerank, synthesize with citation enforcement and prompt caching, guard with multiple safety layers, and monitor drift continuously.

The hard parts are not the model. They are jurisdiction-aware compliance (HIPAA, GDPR, SOC2), citation-level auditability, cost control (prompt caching buys 60-90% discount when hit rate is high), hallucination detection via RAGAS faithfulness, and drift detection when the source documents get updated without the index catching up. This capstone asks you to ship all of it on a 200-question golden set with a red-team suite alongside.

Concept

The pipeline has two sides. Ingestion: docling or Unstructured parses structured documents; ColPali handles visually rich ones; chunks get summaries, tags, and role-based access labels. Vectors go into pgvector + pgvectorscale (under 50M vectors) or Qdrant Cloud; sparse BM25 runs alongside. Conversation: LangGraph handles memory and multi-turn; each query runs hybrid retrieval, reranks with bge-reranker-v2-gemma-2b, synthesizes with Claude Sonnet 4.7 (prompt-cached), passes output through Llama Guard 4 and NeMo Guardrails, and emits a citation-anchored response.

The eval stack has four layers. Golden set (200 labeled Q/A with citations) for correctness. Red team (jailbreaks, PII extraction attempts, off-domain questions) for safety. RAGAS for faithfulness / answer relevance / context precision automatically per-turn. Drift dashboard (Arize Phoenix) watching retrieval quality and hallucination score weekly.

Prompt caching is the cost lever. Claude 4.5+ and GPT-5+ support caching system prompts + retrieved context. At 60-80% hit rate, per-query cost drops 3-5x. The pipeline must be designed for stable prefixes (system prompt + reranked context first) to achieve high cache hit rates.

Architecture

documents (contracts, protocols, policies)
      |
      v
docling / Unstructured parse + ColPali for visuals
      |
      v
chunks + summaries + role-labels + jurisdiction tags
      |
      v
pgvector + pgvectorscale  +  BM25 (Tantivy)
      |
query + role + jurisdiction
      |
      v
LangGraph conversational agent
   +--- retrieve (hybrid)
   +--- filter by role + jurisdiction
   +--- rerank (bge-reranker-v2-gemma-2b or Voyage rerank-2)
   +--- synthesize (Claude Sonnet 4.7, prompt cached)
   +--- guard (Llama Guard 4 + NeMo Guardrails + Presidio output PII scrub)
   +--- cite + return
      |
      v
eval:
  RAGAS faithfulness / answer_relevance / context_precision (online)
  Langfuse annotation queue (sampled)
  Arize Phoenix drift (weekly)
  red team suite (pre-release)

Stack

Ingestion: Unstructured.io or docling for structured documents; ColPali for visually-rich PDFs
Vector DB: pgvector + pgvectorscale under 50M vectors; Qdrant Cloud otherwise
Sparse: Tantivy BM25 with field weights
Orchestration: LlamaIndex Workflows (ingestion) + LangGraph (conversation)
Re-ranker: bge-reranker-v2-gemma-2b self-hosted or Voyage rerank-2 hosted
LLM: Claude Sonnet 4.7 with prompt caching; fallback Llama 3.3 70B self-hosted
Eval: RAGAS 0.2 online, DeepEval for hallucination and jailbreak suites
Observability: Langfuse self-hosted with annotation queue; Arize Phoenix for drift
Guardrails: Llama Guard 4 input/output classifier, NeMo Guardrails v0.12 policy, Presidio PII scrub
Compliance: role-based access labels on chunks; jurisdiction tags for GDPR/HIPAA

Get it reviewed

Paste your capstone report. Jev scores the evidence against each rubric line; the final mark comes from your mentor’s code review.

Paste your evidence: the README section, results tables, numbers, test output

Reviews open with an access code.

Adapted from open-source work: AI Engineering from Scratch by Rohit Ghumare (MIT, lesson text, code and quizzes); Awesome LLM Apps by Shubham Saboo (Apache-2.0, starter code); AI Engineering Interview Questions by Outcome School (Apache-2.0, interview questions). Capstones, milestones, data packs and Zambian context by Zambrite.