Preparing your design workspace

Loading the problem and your latest saved attempt…

Design Observability for AI Agent Tool Calls

Trace agent tool calls, failures, exceptions, and retries through a privacy-safe, high-volume observability platform.

Problem statement

For this exercise, assume an AI agent invokes registered external tools and emits structured telemetry for the agent span, each logical tool call, and every physical attempt.

An operator must search recent failures and open one trace to understand the complete agent-to-tool chain, redacted input and output summaries, status or exception evidence, and why a retry occurred. Every trace, span, logical tool call, attempt, and telemetry event has a stable identity.

Delivery is at least once, so duplicate events must not create duplicate spans, attempts, or counts; out-of-order events and unresolved parents must remain repairable and visible. Agents must continue operating during an observability outage.

Local collectors buffer for up to fifteen minutes, while acknowledged telemetry must survive one availability-zone failure and become searchable asynchronously. Store only bounded structured metadata and redacted summaries; raw secrets and unrestricted payloads are outside scope.

Enforce tenant isolation, expose ingestion and search lag, and mark sequence gaps or incomplete traces rather than treating missing telemetry as success.

Instrumenting tool implementations, changing agent retry policy, executing tools, training models, and storing full prompt or tool payload bodies are outside the platform boundary.

What to build

  1. Agent and collector instrumentation can durably submit bounded structured span, tool-call, attempt, status, exception, and retry telemetry with stable identities and explicit per-event outcomes.
  2. An authorized operator can retrieve one agent-run trace and drill through parent spans, logical tool calls, physical attempts, redacted input and output summaries, failures, exceptions, and retry links.
  3. An authorized operator can search a bounded tenant and time window by tool, outcome, error class, and retry result with deterministic pagination and explicit freshness.
  4. Operators can inspect durable ingestion position, searchable watermark, backlog, rejected or conflicting events, redaction failures, emitter sequence gaps, and partial-trace status.

At scale

  • For this exercise, assume a peak of 100,000 tool-call attempts per second, with bounded structured telemetry for every reported agent span and tool attempt.
  • For this exercise, make accepted failure and retry telemetry searchable within five seconds under normal load.
  • For this exercise, retain structured trace evidence in hot searchable storage for seven days and in archive storage for thirty days.
  • For this exercise, local collectors buffer for up to fifteen minutes, acknowledged telemetry survives one availability-zone failure, and one regional collector outage does not stop healthy regions from accepting events.