LLM Observability and Alerting Platform
Design trace, metric, alerting, dashboard, and usage-billing paths for high-volume LLM applications.
Problem statement
For this exercise, assume SDKs emit nested LLM traces containing spans, token counts, model identity, latency, errors, tags, and tenant metadata.
The platform ingests traces globally, powers near-real-time dashboards, evaluates rules such as N failures in M minutes, delivers alerts reliably, detects anomalies, and produces auditable monthly usage totals. Events may be late or duplicated. Exclude prompt experimentation UI and model hosting.
What to build
- Accept nested trace and token events with tenant isolation, deduplication, and schema evolution.
- Serve trace search, metric dashboards, and release comparisons with explicit freshness.
- Evaluate threshold and anomaly rules, deliver notifications reliably, and close auditable monthly usage ledgers.
At scale
- Ingest 2 million spans per second from three regions with bursts ten times higher.
- Expose aggregate dashboards within 30 seconds and searchable traces within 2 minutes.
- Evaluate alert windows every minute and deliver accepted alert occurrences within 60 seconds.
- Close monthly billing from immutable usage events while allowing bounded late corrections.