Design Distributed LLM Embedding and Output Sharding
Shard very large input-embedding and language-model output matrices while controlling memory, communication, skew, and sampling cost.
Problem statement
Design the distributed execution path for an LLM whose input embedding matrix [vocabulary, hidden] and output language-model head [hidden, vocabulary] do not fit on one accelerator. Compare vocabulary-dimension and hidden-dimension sharding.
Explain token routing, collective communication, skew, local and global top-k sampling, placement, failure behavior, and how the chosen sharding composes with the surrounding tensor-parallel layers.
For this exercise, assume one model replica spans 64 accelerators, the vocabulary contains 20 million entries, hidden width is 16,384, and online decoding must preserve one pinned model and sharding version per request. Model training, tokenizer design, and model-quality evaluation are outside scope.
What to build
- The runtime can transform token IDs into distributed hidden representations for a pinned model version.
- The runtime can compute output candidates without materializing the entire vocabulary on every accelerator.
- The runtime can produce exact global top-k or sampled next-token results under the declared policy.
- Operators can load, drain, replace, and observe a complete shard group without mixing model generations.
At scale
- The source explicitly considers input embedding [V, H] and output head [H, V] sharded by vocabulary or hidden dimension.
- For this exercise, assume V is 20 million, H is 16,384, and one serving replica spans 64 accelerators.
- For this exercise, target 20 milliseconds p95 for embedding lookup and output-head plus sampling work per decode step, excluding transformer layers.
- For this exercise, accelerator or collective failure aborts the affected request rather than returning an approximate token from an incomplete shard group.