Enterprise Multi-Model AI Gateway
An OpenRouter-inspired final capstone: design and build the control plane, request plane, routing policy, reliability layer, enterprise tenancy, and economics behind a unified multi-model AI gateway. Built with mock providers from public documentation - it does not claim access to OpenRouter's private systems.
The promise
a normalized endpoint that hides provider-specific wire formats
a versioned registry of capabilities, regions, prices, and health
filter by capability, region, data policy, allowlist, budget, and health
pre-token failover; post-token error, never silent replay
per-workspace budgets, concurrency, priority, fencing
append-only idempotent ledger, deterministic settlement
SSO, guardrails, audit, SLAs, invoicing boundaries
The capstone question
Design an enterprise AI gateway that lets one application call hundreds of models through one API while enforcing tenant policy, selecting a compliant provider, surviving provider failures, accounting for every token, and producing an auditable operating picture.
Minimum implementation
Build a local gateway simulator with at least three mock providers (cheap/slow, expensive/fast, unreliable/policy-incompatible). Implement normalized requests, capability/policy filtering, explainable versioned routing, stream-aware fallback, per-workspace budgets and per-tenant concurrency admission, an idempotent usage/cost ledger, and PII-safe traces. Implement at least three of the seven components from scratch; the rest may use small standard-library helpers.
Failure-injection matrix
| Failure | Expected behavior |
|---|---|
| Provider timeout before tokens | Retry/fail over within bounded retry budget |
| Provider disconnect after first token | Do not transparently replay; emit stream error and partial-attempt event |
| No provider meets ZDR/region policy | Fail closed; explain rejected candidates |
| Duplicate idempotency key | Return/reconcile the original logical request according to contract |
| Workspace budget exhausted | Reject before provider dispatch |
| Tenant concurrency saturated | Queue, reject, or shed according to priority policy |
| Leader lease expires mid-admission | Fencing prevents stale owner from committing admission |
| Provider price changes | New version applies to new decisions; old ledger remains reproducible |
| Malformed provider usage | Quarantine settlement and alert; never invent silent cost |
| PII appears in payload | Redact or reject before durable trace emission |
Staff defense
- Why is the gateway valuable if providers already expose APIs?
- What must be strongly consistent, and what can be eventually consistent?
- How do you prove a fallback did not violate data policy?
- When is retrying a stream unsafe?
- How do you prevent one tenant from buying all provider capacity?
- How do you reconcile provider usage with customer invoices?
- Which features should be bought from an identity, billing, or observability vendor?
Self-grading rubric - 24 points
Score each dimension 0–2 across product boundary, API normalization, registry, routing, data policy, reliability, tenancy, accounting, observability, security, benchmark, and staff defense. A passing capstone requires 18/24, no zero in policy, reliability, tenancy, or accounting, and a complete evidence bundle.
Boss extensions
Multi-region active-active routing · contextual bandit routing with rollback · dedicated-capacity scheduler · BYOK key isolation · tool-call-aware routing · invoice reconciliation · OpenTelemetry-style trace broadcasting · policy-aware semantic cache.
It takes vector.dot(), retrieval, queues, traces, cost attribution, inference scheduling, and leader election - and turns them into a product someone could actually buy.