I4Intermediate 45 minutes reading and codeLocked

Caching, Rate Limiting & Cost Optimization

Most AI startups do not die from bad models. They die from bad unit economics. A single GPT-4o call costs fractions of a cent. Ten thousand users making ten calls per day costs $250 in input tokens alone -- before you charge a single dollar. The companies that survive are the ones that treat every API call as a financial transaction, not a function call.

In this lesson

  1. 01Learning Objectives
  2. 02The Problem
  3. 03The Concept
  4. 04Build it
  5. 05Use it
  6. 06Exercises
  7. 07Key Terms
  8. 08Further Reading

About 4900 words of reading, with code, 5 quiz questions and the tutor.

The full lesson opens with an access code

You can see what this lesson covers and the milestone it sets. The full text, code, quizzes, the tutor and the AI reviews come with the K850 practice, along with weekly build sessions and code review.

Codes come with the K850 practice. Book a place

Book the K850 practiceRead an open lesson

Milestone forA production labour-compliance assistant

Know the cost of every answer in kwacha

Log tokens, model and cache status for every call and convert to ZMW at a dated rate. Add response caching for repeated questions and routing of easy questions to a cheaper model, and prove quality holds on the golden set.

Done when

  • ZMW per query and per role from logged tokens at a dated exchange rate
  • A response cache with a written invalidation rule
  • Routing that cuts cost by at least 30% with golden-set scores within 2 points

Stretch: Project monthly cost for 50, 500 and 5,000 users

Reviews open with an access code.

Interview practice

Questions on this topic that AI engineering interviews ask, with the companies reported to ask them. Answer the way you would out loud; Jev scores it and DeepSeek tells you what to add.

Scoring opens with an access code. You can still read the questions and prepare.

I3Guardrails, Safety & Content FilteringNext · I5Prompt Caching and Context Caching

Adapted from open-source work: AI Engineering from Scratch by Rohit Ghumare (MIT, lesson text, code and quizzes); Awesome LLM Apps by Shubham Saboo (Apache-2.0, starter code); AI Engineering Interview Questions by Outcome School (Apache-2.0, interview questions). Capstones, milestones, data packs and Zambian context by Zambrite.

Chat on WhatsApp