AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

A gateway sits between your apps and model providers. Core features are provider routing, fallback, retries, rate limiting, secret references, observability, guardrails. Market split in 2026: **LiteLLM** is MIT OSS with 100+ providers, OpenAI-compatible, but breaks down around ~2000 RPS (8 GB memory, cascading failures in published benchmarks); best for Python, <500 RPS, dev/prototyping. **Portkey** is control-plane-positioned (guardrails, PII redaction, jailbreak detection, audit trails), went Apache 2.0 open-source March 2026, 20-40 ms latency overhead, $49/mo production tier. **Kong AI Gateway** built on Kong Gateway — Kong's own benchmark on same 12 CPUs: 228% faster than Portkey, 859% faster than LiteLLM; $100/model/month pricing (max 5 on Plus tier); enterprise-fit if you're already on Kong. **Bifrost** (Maxim AI) — automatic retries with configurable backoff, fallback to Anthropic on OpenAI 429. **Cloudflare / Vercel AI Gateways** — managed, zero-ops, basic retry. Data residency drives the self-host decision; Portkey and Kong sit in the middle with OSS + optional managed.

In this lesson

01Learning Objectives
02The Problem
03The Concept
04Use it
05Exercises
06Key Terms
07Further Reading

About 1200 words of reading, with code, 6 quiz questions and the tutor.

The full lesson opens with an access code

You can see what this lesson covers and the milestone it sets. The full text, code, quizzes, the tutor and the AI reviews come with the K850 practice, along with weekly build sessions and code review.

Book the K850 practice Read an open lesson

Milestone forA production labour-compliance assistant

Put a gateway in front of the models

Route all model calls through a gateway with a primary and a fallback provider, per-role budgets, rate limits and caching. Simulate a provider outage and show the assistant keeps answering.

Done when

All model calls go through a gateway with primary and fallback providers
Per-role budgets and rate limits enforced
An outage drill: primary disabled, answers continue, failover visible in traces

Stretch: Add semantic caching at the gateway and measure hit rate

Paste your evidence: the README section, results tables, numbers, test output

Reviews open with an access code.

Interview practice

Questions on this topic that AI engineering interviews ask, with the companies reported to ask them. Answer the way you would out loud; Jev scores it and DeepSeek tells you what to add.

Your answer

Scoring opens with an access code. You can still read the questions and prepare.

Adapted from open-source work: AI Engineering from Scratch by Rohit Ghumare (MIT, lesson text, code and quizzes); Awesome LLM Apps by Shubham Saboo (Apache-2.0, starter code); AI Engineering Interview Questions by Outcome School (Apache-2.0, interview questions). Capstones, milestones, data packs and Zambian context by Zambrite.