Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Product Feature Event Analytics Platform

Design an extensible event platform for distinct-user analysis, bounded raw troubleshooting, most-viewed URL ranking, and subscription-conversion measurement.

Problem statement

Design an internal product-feature instrumentation and analytics platform. Web features emit JSON events with common source and type fields plus evolving feature-specific key-values; for example, a Doc Chat event may include question and answer.

Traffic is about 10 events per second today and may grow to 100 to 200 events per second. Internal queries should return within seconds.

The platform must count unique users over a time window with optional source and type filters. It must also let an authorized analyst inspect bounded raw records using feature-specific predicates, such as Doc Chat events whose answer is null or shorter than 10 characters.

The design should explain how events reach a durable topic, how processors scale, how the schema stays extensible, how data is retained, and which one operational or analytical storage direction owns supported queries.

Choose and defend that primary direction; any derived structure must have a named owner and rebuild boundary rather than becoming a second source of truth.

For this exercise, assume each event has a stable event_id, pseudonymous user_id, occurred_at, source, type, schema_version, and a bounded JSON attributes object. Duplicate event_id deliveries count once, and corrections append a new record rather than mutating accepted raw evidence in place.

Raw events remain queryable for 90 days, daily aggregate sketches remain for two years, late events up to 24 hours are accepted, 50 internal analysts issue up to 20 concurrent queries, and a raw query returns at most 10,000 rows.

A distinct-user query whose entire half-open range is within raw retention supports arbitrary timestamp boundaries and returns an exact count.

A query that includes older retained history must use whole UTC-day boundaries and returns the union of daily mergeable sketches with relative standard error at most 1%; a non-day-aligned old-history window is rejected instead of silently counting full days.

Ingestion is authenticated, attributes are allowlisted by source and schema version, direct personal identifiers are rejected, analyst access is audited, and sensitive free text is encrypted and redacted from ordinary logs.

The interview report additionally emphasizes high-concurrency, high-volume ingestion and explicitly asks about topics, partitions, consumers, and consumer groups.

For this exercise, a scale follow-up raises the peak to 100,000 accepted events per second with bursts to 200,000, while query and retention semantics remain unchanged.

Design ingestion, validation, durable transport, partitioning, consumer-group ownership and rebalance behavior, processing, raw and aggregate storage, query contracts, schema evolution, retention, privacy, late-data correction, overload control, observability, and recovery.

End-user dashboards, experimentation assignment, recommendation models, and arbitrary public SQL are outside scope.

The platform must also answer two web-application analytics questions: the most-viewed URLs in a bounded time window and the subscription-conversion rate for an explicitly defined eligible cohort. The source names those questions but not their semantics.

Before choosing storage, define URL canonicalization, whether a view means all accepted page-view events or distinct viewers, top-K and tie ordering, the conversion numerator and denominator, attribution window, event-time boundary, late-data behavior, and how corrections revise a result.

Do not silently choose denominator, identity, or cross-device attribution semantics.

What to build

  1. Accept authenticated, source-versioned product events with an allowlisted common envelope; deduplicate redelivery, retain and revise late data under explicit policies, and protect sensitive attributes with authorization and audit.
  2. Serve bounded distinct-user counts and authorized raw-event pages with explicit window, exact-or-estimated, granularity, pagination, freshness, retention, and unsupported-range outcomes.
  3. Return a stable top-K ranking of canonical URLs for a bounded event-time window under an explicit view metric, tie rule, freshness contract, and late-data revision policy.
  4. Return subscription-conversion rate for an explicitly defined eligible cohort, conversion event, attribution window, numerator, denominator, and correction policy.

At scale

  • The source reports about 10 events per second today and growth to roughly 100 to 200 events per second.
  • For this exercise, assume 90 days of queryable raw data, two years of daily mergeable aggregate sketches with at most 1% relative standard error, and events arriving up to 24 hours late.
  • For this exercise, distinct-user windows wholly within raw retention may use arbitrary timestamp boundaries and return exact counts; a window that includes older retained history must be aligned to whole UTC days and returns an estimated sketch union, while an unaligned old-history window is rejected.
  • For this exercise, assume 50 authorized analysts issue at most 20 concurrent queries, each bounded raw query returns at most 10,000 rows, and supported queries complete within seconds.
  • For this exercise, assume feature teams add source-versioned attribute schemas continuously without requiring a central storage migration for each new key.
  • Source-backed query families additionally include most-viewed URL ranking and subscription-conversion rate; their metric and attribution semantics are unspecified and must be made explicit rather than inferred.
  • For this exercise, the high-volume follow-up raises accepted ingestion to 100,000 events per second with bursts to 200,000; consumers must rebalance and recover without dropping or applying an accepted event twice.