3 Rounds TikTok AI Infra Internship Interviews

First Round

Project Experience

1. Introduce the research work on Efficient Transformers and explain the main contributions.

Fundamentals

1. What are the three main components of a CPU cache line, and what does each component do?
2. Why do modern CPUs use multiple levels of cache?
3. What happens at the hardware level when a cache miss occurs?
4. What performance overhead does the page table mechanism introduce, and how can it be reduced?
5. What are the benefits of using page tables / virtual memory?
6. How does function overloading in C++ work, and how does the compiler implement it?
7. What are atomic operations in C++, and why are they needed?

Coding

1. Board Connectivity: Given a board, determine whether there is a connected path from the first row to the last row.

Second Round

Project Experience

1. Deep dive into the research on Efficient Transformers, including the approach, implementation, and key findings.

Fundamentals

1. How would a Sparse Attention kernel be implemented in Python?
2. What are the common parallelism strategies for distributed training of large models?
3. How can high GPU memory usage during model training and inference be reduced? Discuss approaches such as sparsification, gradient checkpointing, quantization, mixed-precision training, and knowledge distillation.
4. How would GPU parallel computing code be written and implemented?

Coding

1. Given a complete Graph class, implement topological sort.

Third Round

Project Experience

1. Explain the key innovations behind the proposed Sparse Attention mechanism.
2. How effective are techniques such as Efficient Attention, Mixture of Experts (MoE), and distributed training in real-world production systems?

Fundamentals

1. In C++ polymorphism, when are the function addresses in a virtual function table (vtable) determined? When is an object's virtual table pointer (vptr) initialized?
2. How can the memory allocated by a C++ vector be released?
3. Design a stack that supports all standard stack operations and can also return the current minimum element in O(1) time.

Coding

1. Next Permutation — LeetCode 31