Design a Privacy Registry Check Before Processing
Add a check against an outside government registry to an existing request-processing service so that nobody listed in the registry has their data processed, even when the registry is slow, rate-limited or down.
Problem statement
Design a change to an existing service: before it processes anyone's data, it must check that person against an outside privacy registry.
For this exercise, assume the service receives requests from partner organizations. Each request is about one person and carries that person's government-issued identifier along with the data to process.
Today an intake API stores each request and answers the partner within a second, a queue holds the work, and workers process each request within 4 hours. Partners read a request's outcome through the existing status API.
For this exercise, also assume a government agency runs a registry of people whose data must not be processed. Its API accepts up to 100 identifiers per call and returns each listed identifier paired with its registry ID; an empty answer means none of them are. Every answer carries a response ID and a time.
People can be added to the registry at any time, so an answer may be relied on for at most 24 hours. If a person is listed, their request must not be processed: its data is deleted, only a minimal record of the decision is kept, and the partner is told the request could not be processed, without being told why.
The registry is outside your control. For this exercise, assume it allows at most 10 calls a second from your service, answers most calls within a second, is down for planned maintenance of up to 4 hours once a month, and sometimes fails without warning.
A request must never be processed without a registry answer from the last 24 hours that does not list the person. Partners have agreed that requests may take longer than 4 hours while the registry is down.
You design where the check goes, what is stored, and how the service behaves when the registry is slow or unavailable. The processing itself, and data processed before the check existed, are out of scope.
What to build
- Every request is checked against the registry before processing, and no worker can process a request without a registry answer from the last 24 hours that does not list the person.
- A request for a listed person is never processed: its data is deleted within an hour and the partner sees 'could not be processed' without the reason.
- While the registry is slow or down, the intake API still accepts requests and they wait unprocessed until they can be checked; a request that cannot be checked within 72 hours is returned to the partner to resubmit and its data is deleted.
- Every allow or block decision is recorded with the registry's response ID and time, so the service can show which check allowed each processed request, while keeping no more about a listed person than that record.
At scale
- For this exercise, assume about 3 million requests a day, about 35 a second on average and up to about 150 a second at peak, about 1 million different people a day.
- For this exercise, assume the registry allows at most 10 calls a second with up to 100 identifiers each, answers most calls within a second, and is unavailable for up to 4 hours of planned maintenance a month plus occasional unplanned outages.
- For this exercise, assume requests are still processed within 4 hours of intake while the registry is healthy, and the intake API keeps answering partners within a second at all times.