TikTok | AI Infra Intern | Onsite Round 2

Implement Linear Attention from scratch

• Write a basic implementation of Linear Attention without relying on existing high-level implementations.

Implement MoE from scratch

• Implement a Mixture of Experts (MoE) layer, including the routing and expert computation.

Implement MHA from scratch

• Implement Multi-Head Attention (MHA) from scratch.

Implement MLA from scratch

• Implement Multi-Head Latent Attention (MLA) from scratch.

Memory access and computation analysis

• For each of the implementations above, calculate and analyze the amount of computation and memory access required.

DeepSeek-V3 architecture

• Write pseudocode for the DeepSeek-V3 architecture from scratch.
• Manually calculate the number of model parameters.
• Manually estimate the amount of memory access required for each inference pass.

Coding Question: Heap Sort

• Implement Heap Sort from scratch.

Implement and optimize Reduce

• Implement a Reduce operation from scratch.
• Discuss and implement possible optimizations to improve its performance.