Design a Real-Time VWAP Analytics Service
Turn live trade ticks from 100 exchanges into an up-to-date volume-weighted average price for every listing, push it to screen users within seconds, and answer enterprise queries with values that say whether any trades may be missing.
Problem statement
For this exercise, assume a market-data company wants to calculate the volume-weighted average price (VWAP) of every listed stock from live trades, and to serve it to two kinds of users.
The VWAP of a listing over a window is the sum of price times quantity over every trade in that window, divided by the sum of the quantities. A listing is one symbol on one exchange.
Trades arrive as ticks from 100 exchanges. Each exchange sends its own message format, numbers its messages in order on each of its feed channels (the separate message streams an exchange sends), and may repeat or drop messages. A dropped range of messages can be fetched again from that exchange's replay service.
A tick describes one trade: the symbol, the price, the quantity and the exchange time.
For every listing the service keeps two values: the VWAP since that exchange opened today, and the VWAP over the last 5 minutes. Screen users see live values for the listings on their screens, pushed to them as values change.
Enterprise users ask for values through an API, and they care most that a value includes every trade, so a reply must tell them when a value may be missing trades.
Also assume that each trade must count exactly once, and that prices and quantities are exact decimals. Trade corrections and cancellations, quotes, other windows, combining one symbol across exchanges, and any history before today are out of scope.
What to build
- Take trade ticks from all 100 exchanges, each using a different message format, turn them into one common trade record, and count each trade exactly once even when a feed repeats or drops messages.
- Keep the day VWAP and the 5-minute VWAP of every listing up to date, refreshed at least once a second for every listing that traded.
- Push changed VWAP values to each screen user for the listings on their screen, no more than 2 seconds behind the latest trade.
- Let enterprise users query current VWAP values through an API, with each value marked as complete or as possibly missing trades.
At scale
- For this exercise, assume 100 exchanges with between 100,000 and 1,000,000 listings each, that a busy listing trades about 100 times a second, and that one decoded tick is about 20 bytes.
- For this exercise, assume most listings trade far less often than busy ones, so all exchanges together peak at about 5 million ticks a second.
- For this exercise, assume about 100,000 screen users, each watching up to 50 listings, whose values must be at most 2 seconds behind the latest trade, and about 1,000 enterprise users who together make up to 2,000 API queries a second, each for up to 100 listings.
- For this exercise, assume any VWAP calculation server can fail at any moment, and that after such a failure every value must be correct again within 1 minute, with no trade counted twice or lost.