Design a Website Content Parsing Service
Design a bounded service that turns mixed website content into traceable parsed blocks and context-preserving chunks.
Problem statement
Design the architecture of a website parsing service for pages containing images, large PDFs, and large amounts of text. Explain how a PDF around 100 MB can be processed without losing important context, including chunking, overlap, and possible LLM use.
For this exercise, assume one authorized parse job starts from one URL, fetches that page and its directly embedded supported assets without discovering further links, and publishes one immutable parsed-content generation.
Also assume deterministic extraction remains authoritative while optional LLM annotations may enrich but never silently replace it. Search, retrieval, question answering, model training, and general web crawling are out of scope.
What to build
- For this exercise, accept an idempotent authorized parse job for one URL and expose its progress, terminal outcome, warnings, and output generation.
- Extract ordered text blocks, document structure, image metadata and references, and supported PDF content into an immutable source-linked manifest.
- Produce traceable chunks for large text and PDFs while retaining page, section, neighborhood, and source-version context.
- For this exercise, publish a parsed generation atomically only after required stages finish, with explicit partial, unsupported, and failed-asset outcomes.
At scale
- The reported focus includes images, large amounts of text, and PDFs around 100 MB that must be processed without losing important context.
- For this exercise, assume 100 accepted parse jobs per second at peak, 10,000 concurrently active jobs, and no more than 500 MB of fetched bytes per job.
- For this exercise, make 99 percent of healthy jobs containing a 100 MB PDF terminal within five minutes while keeping each worker's memory bounded below the full job size.
- For this exercise, retain immutable source bytes and parsing manifests for 30 days, survive one process or availability-zone failure after acknowledgement, and never expose a partially published generation.