How large models are trained on many GPUs: distributed training explained

No single GPU can train a large language model. How data, tensor and pipeline parallelism split the work, what ZeRO and FSDP do, and why communication between GPUs becomes the real bottleneck.

Large models are trained by splitting work across many GPUs. Data parallelism gives each GPU a copy of the model and different data; tensor parallelism splits individual layers across GPUs; pipeline parallelism splits the model’s layers into stages. Techniques such as ZeRO and FSDP shard parameters and optimiser states so models too big for one GPU can still be trained.

Distributed training sounds like a topic only frontier labs need, but its ideas show up whenever a model is too big or a dataset too large for one machine. Understanding it also explains why training large models is so expensive.

Why can one GPU not train a large model?

Training needs memory for the parameters, their gradients, the optimiser’s extra state and the activations from the forward pass. For large models this adds up to far more than one GPU holds. Even when it fits, training on one GPU would take far too long. Splitting the work solves both problems, at the cost of GPUs having to communicate constantly.

What are the main kinds of parallelism?

Large training runs combine several of these, often called 3D parallelism. Megatron-LM popularised tensor and model parallelism for transformers.

What do ZeRO and FSDP do?

In plain data parallelism, every GPU stores a full copy of parameters, gradients and optimiser states, which wastes memory. ZeRO, used in DeepSpeed, shards these across GPUs in stages, so each GPU stores only a slice and fetches the rest when needed. PyTorch’s Fully Sharded Data Parallel (FSDP) applies the same idea natively. Together they made training much larger models on the same hardware practical.

What other techniques save memory and time?

Why is communication the bottleneck?

GPUs are extremely fast at arithmetic but must constantly exchange gradients and activations. If the network between them is slow, they spend more time waiting than computing. That is why large training clusters use very fast interconnects, and why choosing the right mix of parallelism depends on the hardware layout as much as on the model.

How do engineers decide model and data size?

Scaling research, such as the Chinchilla findings that model size and training data should grow together for a fixed compute budget, guides these choices. Engineers estimate cost and time before committing to a run, test settings in small ablations, and only then launch the full job. Our walkthrough of how small language models are trained covers the training pipeline these techniques serve.

What does a real training run involve beyond the GPUs?

A large run is as much an operations project as a modelling one. Data must be cleaned, deduplicated, filtered and tokenised at scale before training starts. Jobs are launched through schedulers, monitored continuously for loss spikes and hardware failures, and checkpointed regularly. Teams run small ablations first to choose settings, then document every run so results can be compared and reproduced. These habits are what separate a working training pipeline from an expensive failure.

Which skills does distributed training build?

Can you learn distributed training without a big cluster?

Yes. The concepts and code can be learned with two GPUs, or even simulated on one, and cloud providers rent multi-GPU machines by the hour. Learning to run data-parallel and FSDP training on a small model teaches the same patterns used at scale. Containerised environments and job schedulers, covered in our Docker and Kubernetes guide, are part of the same toolkit.

Where can you learn this properly?

Program Zero’s advanced LLM engineering phase covers distributed data parallelism internals, FSDP and DeepSpeed ZeRO stages, tensor and pipeline parallelism, mixed precision, checkpointing and fault tolerance, and scaling laws, after the phase where you train a small GPT on one machine.

Program Zero runs live online, so you can learn this from anywhere in India: metros such as Bengaluru, Pune, Hyderabad and Noida, growing cities such as Lucknow, Kanpur, Visakhapatnam and Madurai, and smaller towns such as Siliguri, Jamshedpur, Durgapur and Cuttack. Every learner joins the same evening classes, at the same fee.

Frequently asked questions

Related reading