Preparing your design workspace

Loading the problem and your latest saved attempt…

Design a Reliable Search Indexing Pipeline

Keep an Elasticsearch index in step with a product database without losing any committed change, while protecting the cluster from slowdowns, outages, and sudden write spikes.

Problem statement

Design the pipeline that keeps an Elasticsearch search index up to date with the changes a product makes to its records, and make it robust enough that no change is ever lost.

For this exercise, assume a product service stores its records, such as catalog items, in a primary database that stays the source of truth. Every create, update, or delete of a record must eventually show up in an index on an existing Elasticsearch cluster that serves the product's search page.

Each record has a stable ID and a version number that goes up with every committed change. You are designing the path from a committed change to the indexed document; the database, the search page, and ranking already exist.

For this exercise, assume the search cluster can become slow, reject requests, or be unavailable for up to 30 minutes, and that write traffic can jump suddenly to ten times its normal rate, for example during a bulk import.

No committed change may be lost, an older change must never overwrite a newer one in the index, and a deleted record must not come back. The index may lag behind the database, but the lag must be visible.

Query serving, ranking, the design of the search document fields, and multi-region replication are out of scope.

What to build

  1. Every committed create, update, or delete of a record is captured and eventually applied to the search index, even if the search cluster is down when the change happens.
  2. Applying a change is safe to repeat: a duplicate or late change never makes an indexed document older than the newest applied version, and an older update never brings back a deleted record.
  3. During a cluster slowdown, an outage, or a sudden write spike, the pipeline slows down, holds the backlog, and catches up on its own, without dropping changes.
  4. Operators can see how far the index lags behind the database, and can replay or reconcile records whose changes failed or drifted.

At scale

  • For this exercise, assume about 50 million records, a normal rate of 2,000 changes per second, and spikes of up to 20,000 changes per second that last up to 15 minutes.
  • For this exercise, assume 99 percent of changes should be searchable within 10 seconds of commit during normal operation.
  • For this exercise, assume the search cluster can be slow or unavailable for up to 30 minutes, and every change committed during that time must still be indexed afterwards.
  • For this exercise, assume the cluster handles bulk requests of a few thousand documents well, while one write request per change at peak rate would overload it.