Design a Redundant UDP Market-Data Ingestion Pipeline
Move a live market-data feed from loader servers to processor servers over redundant UDP copies, and turn those copies into one clean, ordered, stored stream that connected users can follow.
Problem statement
Design a real-time market-data pipeline. Several stock exchanges each publish a continuous stream of price updates. Loader servers read the exchange data and send it on over UDP to a group of processor servers. Every processor must receive the data of every loader.
UDP does not promise delivery, so packets can be lost, arrive twice, or arrive out of order. The processors must still end up with every message once and in order, store the data, and serve the users who are connected to the system.
For this exercise, assume you control the software on the loaders and the processors, so you may add fields to messages and add services of your own.
Assume each loader is responsible for a fixed set of instruments, all updates for one instrument come from one loader, and a loader keeps its most recent messages in memory for a while but cannot keep everything.
Also assume each loader sends several copies of every message to every processor, over separate network paths. Out of scope: the connections to the exchanges, loader failure and replacement, user login, entitlements and billing, and order entry.
What to build
- Each message a loader sends is applied once and in the order the loader sent it, even though several copies of it arrive, copies are duplicated or reordered, and some copies are lost.
- When every copy of a message is lost, the processor notices the gap and gets the message back from the loader, or marks the affected instruments as stale if it cannot in time.
- Every accepted message is stored for 7 days, and the latest state of each instrument can be read, so a processor that restarts catches up without applying anything twice.
- Connected users receive current updates for the instruments they follow, and a slow user never slows the pipeline or other users.
At scale
- For this exercise, assume 3 exchange feeds are read by 4 loaders, which send to 6 processors. At peak the loaders produce 300,000 messages per second in total, about 100 bytes each, and about 100,000 per second on average over a trading day. Together they cover about 100,000 instruments.
- For this exercise, assume each loader sends 3 copies of every message to every processor, over 3 separate network paths. Treat one packet as one message. A path loses about 0.1% of its packets on average, and can lose bursts of up to 50 packets in a row.
- For this exercise, assume a user must see an update within 1 second of the loader sending it, at the 99th percentile, or be told that the instrument is stale. Every delay on the way counts, including any wait to repair a loss and any time spent merging updates for users.
- For this exercise, assume 50,000 users are connected at peak, each following about 20 instruments. A user needs only the latest value of an instrument, so updates for one instrument may be merged, but a user must not be sent more than two updates per second for each instrument.
- For this exercise, assume a restarted processor must serve correct data again within 30 seconds, and the system keeps 7 days of accepted messages for replay and audit. Assume each processor has a 10 gigabit per second network link to the loaders.