Distributed Training Glossary: 16 Terms You Will Hear in Every AI Infra Meeting

Why This List#

Yesterday’s passage was about serving — KV cache, batching, throughput. This one covers the other half of AI Infra: training at scale. These are the words people in the field actually use every day — in design reviews, in the related-work sections of every paper, and in the post-mortem of every GPU-outage incident. None of them are in a CET-6 word list, all of them are in a random AI Infra meeting.

Each term below comes with a Chinese gloss, an English definition, and a real-world context sentence adapted from the paper or documentation where you would actually meet the term. Sources are listed at the end.

Parallelism: Splitting the Work#

The first question in any training-scale discussion is: does the model fit on one GPU, and if not, how do we split it?

Term	中文释义	English definition	Real-world context
data parallelism (DP)	数据并行	Replicating the full model on every GPU and splitting the batch among them; the GPUs average their gradients with an all-reduce at each step.	”Data parallelism replicates the model on each GPU and shards the data batch across the GPUs — still the most common way to scale training.” (Megatron-LM)
model parallelism (MP)	模型并行	Any scheme that splits the model itself across devices because one GPU cannot hold it; tensor and pipeline parallelism are its two main forms.	”When the model does not fit on a single GPU, model parallelism partitions the model across devices instead of replicating it.” (Megatron-LM)
tensor parallelism (TP)	张量并行	Splitting each layer’s weight matrices across GPUs so that one layer’s computation runs on several devices at once.	”Megatron-LM splits the weight matrices of every transformer layer across 8 GPUs, so each layer is computed jointly by all 8.” (Megatron-LM)
pipeline parallelism (PP)	流水线并行	Splitting the model by groups of layers (“stages”); each stage lives on a different GPU, and activations flow through the stages in sequence.	”GPipe divides a model into layer stages; micro-batches flow through the pipeline one stage after another so that every GPU stays busy.” (GPipe)
ZeRO	零冗余优化器	Sharding the optimizer state, gradients, and parameters across data-parallel ranks so no GPU stores redundant copies — the technique behind DeepSpeed.	”ZeRO eliminates memory redundancies in data- and model-parallel training by partitioning model states across devices.” (ZeRO)
offloading	卸载（把数据放到 CPU 内存）	Moving part of the training state — usually the optimizer state — to host CPU memory when the GPU is full.	”ZeRO-Offload moves the optimizer memory and its computation from the GPU to the host CPU, and performs the weight update on the CPU.” (DeepSpeed)
gradient accumulation	梯度累积	Delaying the optimizer step until gradients from several micro-batches have been summed, to simulate a larger batch size than GPU memory allows.	”With gradient accumulation we update the weights only every N steps, summing the gradients of the previous N mini-batches.” (HF docs)
gradient checkpointing	梯度检查点（重计算）	Discarding most intermediate activations during the forward pass and recomputing them during the backward pass — less memory, more compute.	”Instead of storing every intermediate activation, gradient checkpointing recomputes them in the backward pass, trading compute for memory.” (PyTorch docs)

Communication and Numerics: The Plumbing#

Even with the perfect split, nothing works unless the GPUs can talk to each other — and unless the numbers survive being stored in 16 bits.

Term	中文释义	English definition	Real-world context
all-reduce	全归约	A collective operation that reduces a tensor across GPUs (sum or average) and returns the result to every participant; the final step of every data-parallel update.	”Each data-parallel step ends in an all-reduce: every GPU contributes its local gradient and receives the averaged gradient back.” (NCCL docs)
NCCL	NVIDIA 集合通信库	NVIDIA’s library of collective operations — all-reduce, broadcast, all-gather — tuned for GPU topologies such as NVLink and InfiniBand.	”The NVIDIA Collective Communications Library (NCCL) provides communication primitives for multi-GPU and multi-node communication, optimized for NVIDIA GPUs and networking.” (NCCL)
straggler	掉队的慢节点	The slowest worker in a synchronous run; since all-reduces wait for everyone, one straggler throttles the whole job.	”As the MapReduce paper put it, a straggler is ‘a machine that takes an unusually long time’ to finish its share of the work.” (MapReduce)
mixed precision	混合精度	Training with FP16 (or BF16) forward and backward passes while keeping a master copy of the weights in FP32.	”Mixed precision keeps master weights in FP32, runs the forward and backward passes in FP16, and stores an FP32 copy of the gradients.” (Mixed Precision Training)
bf16	bfloat16	A 16-bit format with FP32’s exponent range but only 7 mantissa bits — the same range as FP32 at half the memory, so it rarely overflows.	”bfloat16 keeps the exponent range of FP32 while cutting the mantissa to 7 bits, which made it the default training format on modern accelerators.” (Bfloat16)
loss scaling	损失缩放	Multiplying the loss by a large factor before backpropagation so that small FP16 gradients do not underflow to zero; the scale is divided back out before the update.	”Loss scaling multiplies the loss by a large factor before the backward pass so that small FP16 gradients do not underflow.” (Mixed Precision Training)
warmup	学习率预热	Starting the learning rate near zero and ramping it up to its peak over the first few thousand steps, which keeps the early updates from destabilizing the weights.	”Goyal et al. proposed linear warmup: increase the learning rate from a small value to its target over the first few epochs, then decay it.” (Large Minibatch SGD)
loss spike	损失尖峰	A sudden jump in the training loss — often the first warning of an unstable learning rate, or of a corrupted batch from the data pipeline.	”A loss spike in the curve is usually the first sign that the learning rate is too high — or that the data loader served a bad batch.” (common usage)

How the Pieces Fit Together#

flowchart LR M["Model doesn't fit on one GPU"] --> TP["Tensor parallel<br/>split each layer's weights"] M --> PP["Pipeline parallel<br/>split layers into stages"] B["Batch is too big / too slow"] --> DP["Data parallel + ZeRO<br/>split data, shard the optimizer state"] TP --> A["8 GPUs cooperate on one training step"] PP --> A DP --> A

In production the three strategies are rarely used alone. A typical 8-GPU node trains with tensor parallelism inside the node (NVLink makes intra-node communication cheap), pipeline parallelism when the model spans many layers, and data parallelism across nodes with ZeRO sharding the optimizer state — a combination colloquially called 3D parallelism. Add mixed precision with bf16 and loss scaling on top, and you have described, in one paragraph, the training setup of most open-weight LLMs. That is the point of this glossary: each word names a knob that a real system has to turn.

Sources#

The context sentences above are adapted (or, where quoted, quoted) from the following real sources:

Megatron-LM: Shoeybi et al., “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism” — https://arxiv.org/abs/1909.08053
GPipe: Huang et al., “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism” — https://arxiv.org/abs/1811.06965
ZeRO: Rajbhandari et al., “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models” — https://arxiv.org/abs/1910.02054
DeepSpeed ZeRO-Offload documentation — https://www.deepspeed.ai/tutorials/zero-offload/
Hugging Face docs, “Methods and tools for efficient training on a single GPU” — https://huggingface.co/docs/transformers/perf_train_gpu_one
PyTorch docs, torch.utils.checkpoint — https://pytorch.org/docs/stable/checkpoint.html
NCCL documentation — https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/operations.html ; NVIDIA NCCL homepage — https://developer.nvidia.com/nccl
MapReduce: Dean & Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” OSDI 2004 — https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf
Micikevicius et al., “Mixed Precision Training” — https://arxiv.org/abs/1710.03740
Wang & Kanwar, “Bfloat16: The Secret to High Performance on Cloud TPUs” — https://arxiv.org/abs/1905.12322
Goyal et al., “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour” — https://arxiv.org/abs/1706.02677

音乐

音乐

Why This List#

Parallelism: Splitting the Work#

Communication and Numerics: The Plumbing#

How the Pieces Fit Together#

Sources#

评论区

音乐

文章目录