Multi-Tenant LLM Trace Forensics
A staff-level system design + coding capstone: build the observability and forensics platform for a large multi-tenant LLM gateway, then answer exact batch forensic queries with Mo’s algorithm.
The scenario
Every request to the gateway produces a structured trace event. Engineers need to answer: between event sequence 1M and 5M, which model had the most errors? For a tenant, what were the distinct failing models in this window? How many tokens were wasted by failed requests in this range?
{
"event_seq": 98231,
"event_id": "evt_8f3a9c",
"trace_id": "tr_9f8s7d",
"tenant_id": "tenant_123",
"model": "llama-70b",
"prompt_tokens": 842,
"completion_tokens": 129,
"latency_ms": 2410,
"status": "error",
"error_code": "CONTEXT_LENGTH_EXCEEDED",
"retriever": "vector_db_a",
"guardrail_triggered": true,
"timestamp": "2026-06-15T14:22:11Z"
} Six logical paths
Events leave the SDK already redacted, validated and deduped, then enter the durable log. Redaction before the log boundary is non-negotiable.
Bounded reorder buffers, watermarks as a function of observed event_seq, monotonic sealing, and correction segments for late arrivals.
Columnar, partitioned by time + tenant + shard, immutable segments, and a metadata catalog with sketch summaries as first-class citizens.
Interactive scan+prune, offline batch with Mo’s algorithm, and approximate sketch execution. One path does not fit all workloads.
Per-tenant scan budgets, weighted admission, circuit breakers around the scanner, and cost attribution back to the tenant.
Query latency, scan volume, cache hit rate, seal lag, reorder occupancy, correction rate, quota consumption - the system observes itself.
The coding problem
Given an immutable array of events and Q offline range queries (l, r), return the model with the most error events in the range (ties → smallest original model_id), its count, the number of distinct error models, and the total wasted tokens. Constraints: N, Q ≤ 200,000; model_id ≤ 10⁹. Required approach: Mo’s algorithm.
events = [
Event(model_id=1, is_error=True, tokens=100),
Event(model_id=2, is_error=False, tokens=50),
Event(model_id=1, is_error=True, tokens=200),
Event(model_id=3, is_error=True, tokens=10),
Event(model_id=2, is_error=True, tokens=70),
Event(model_id=1, is_error=False, tokens=10),
]
queries = [(0, 4), (1, 5)]
# ->
# [ {top_error_model:1, count:2, distinct:3, wasted:380},
# {top_error_model:2, count:1, distinct:2, wasted:80} ] Mo’s reorders the queries so adjacent queries reuse the current range state - O((N+Q)·√N·log N) instead of O(N·Q). The hard part is maintaining the current mode with reversible add/remove, frequency buckets, and tie-breaking on the original ID.
Exact vs approximate mergeability - the killer question
| Statistic | Exact merge from local? | How to merge |
|---|---|---|
| sum | ✅ Yes | Sum of sums |
| distinct_count | ❌ No from local counts alone | Return sets / sorted IDs / Roaring bitmaps, or use HLL |
| mode (top model) | ❌ No from local top-1 alone | Full frequency map per shard, or candidate set + exact recount |
| p95 latency | ❌ No from local p95s | t-digest / DDSketch, merge the sketches |
| heavy hitters | approximate only | Count-Min estimates; Space-Saving finds candidates |
"Just sum the local top-K and pick the global winner." Exact global mode requires full frequency maps or a guaranteed candidate set - local top-K is not enough.
Pitfalls index
Self-grading rubric
26 points across system design (12) and coding (12–14). Target 18+ before scheduling a mock interview. 0–9 not ready; 10–17 getting there; 18–23 solid; 24 hire on the spot.
"Algorithm selection depends on workload shape, rather than problem shape." Mo’s is an offline batch tool - not the interactive default, and not for a single query.