Evaluation & Testing LLM Applications

You would never deploy a web app without tests. You would never ship a database migration without a rollback plan. But right now, most teams ship LLM applications by reading 10 outputs and saying "yeah, looks good." That is not evaluation. That is hope. Hope is not an engineering practice. Every prompt change, every model swap, every temperature tweak changes your output distribution in ways you cannot predict by reading a handful of examples. Evaluation is the only thing standing between your application and silent degradation.

In this lesson

01Learning Objectives
02The Problem
03The Concept
04Build it
05Use it
06Exercises
07Key Terms
08Further Reading

About 5200 words of reading, with code, 5 quiz questions and the tutor.

The full lesson opens with an access code

You can see what this lesson covers and the milestone it sets. The full text, code, quizzes, the tutor and the AI reviews come with the K850 practice, along with weekly build sessions and code review.

Book the K850 practice Read an open lesson

Milestone forA production labour-compliance assistant

The golden set comes before tuning

Build the first 100 questions of the golden set with an HR practitioner: expected answer, expected sections, difficulty. Score retrieval and generation separately and set the baseline every later change must beat.

Done when

100 labelled questions with expected sections, 30 of them reviewed by an HR practitioner
Retrieval and generation scored separately with the scripts in the repo
LLM judge agrees with human verdicts on at least 80% of 30 checks

Stretch: Grow the set to 200 and report the confidence interval on your main score

Paste your evidence: the README section, results tables, numbers, test output

Reviews open with an access code.

Interview practice

Questions on this topic that AI engineering interviews ask, with the companies reported to ask them. Answer the way you would out loud; Jev scores it and DeepSeek tells you what to add.

Your answer

Scoring opens with an access code. You can still read the questions and prepare.

Adapted from open-source work: AI Engineering from Scratch by Rohit Ghumare (MIT, lesson text, code and quizzes); Awesome LLM Apps by Shubham Saboo (Apache-2.0, starter code); AI Engineering Interview Questions by Outcome School (Apache-2.0, interview questions). Capstones, milestones, data packs and Zambian context by Zambrite.