Kubernetes GPU Workload Control Plane
Design a Kubernetes-based GPU control plane that understands NCCL topology, MIG slices, node health, draining, eviction, and job resume.
Problem statement
For this exercise, assume customer clusters run distributed ML workloads through Kubernetes. Custom device plugins advertise full GPUs and MIG slices; scheduling must honor NCCL and network topology.
Health agents detect degraded accelerators or fabric, cordon and drain nodes, preserve checkpoint-capable jobs, and safely replace capacity. The control plane is highly available and reconciles desired versus observed state. Exclude reimplementing Kubernetes and GPU drivers.
What to build
- Project verified GPU, MIG, topology, and health state into schedulable resources.
- Coordinate gang placement and launch for topology-sensitive Kubernetes workloads.
- Quarantine bad capacity, checkpoint or evict workloads, and resume them on fenced healthy resources.
At scale
- Operate 1,000 Kubernetes clusters and 100,000 GPUs across multiple regions.
- Reflect critical node-health changes in scheduling within 10 seconds.
- Support jobs from one GPU to 2,048-GPU gangs and several MIG profiles.
- A quarantined or generation-stale device must never be allocated to new work.