Concept Fable

ZeRO

Conclusion first:Zero Redundancy Optimizer, ZeRO is an optimization strategy that reduces memory redundancy in data-parallel training. By sharding optimizer states, gradients, and model parameters, it avoids every GPU having to store all training states in full, thereby significantly reducing memory usage. ZeRO is usually divided into different stages that progressively reduce redundancy. It is closely related to frameworks such as DeepSpeed and is suitable for large-model training and fine-tuning, but it introduces complexity in communication, checkpoints, recovery, and tuning.

Zero Redundancy Optimizer · Distributed training

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The accounting house no longer makes every apprentice memorize the entire ledger

The key here is to apply the concept's constraints to specific objects, rather than merely describing an outcome.

As scale and exceptions increased, the methods previously held together by ad hoc coordination began to expose blind spots.

Short-term patches restore surface order, but do not address what the concept truly constrains.

The old craftsman turned to recording conditions, actions, results and the chain of responsibility, so that subsequent adjustments had evidence to rely on.

The mountain chief split the ledger into several volumes, and each person memorized only one volume; when a question required someone else's volume, they borrowed it temporarily, finished the calculation, and returned it. This step was troublesome and at first even slowed the schedule; but after several rounds, there was less rework, and the masters no longer changed the rules on a whim. Everyone finally saw that ZeRO trades communication and scheduling complexity for lower memory redundancy, and whether that is worthwhile must be measured in practice by stage and hardware.

Only after saving, borrowing, and merging are clearly distinguished can the team trade communication cost for greater model capacity.

As the team went farther, it also learned to adjust the division of labor when short-staffed, detouring or when passing scrolls slowed down. From then on, the great ledger did not have to rest entirely on everyone's shoulders; only by clearly separating storage, borrowing and consolidation could the team bear larger problems.

Concept Anchor

This story is about: ZeRO

Zero Redundancy Optimizer, ZeRO is an optimization strategy that reduces memory redundancy in data-parallel training. By sharding optimizer states, gradients, and model parameters, it avoids every GPU having to store all training states in full, thereby significantly reducing memory usage. ZeRO is usually divided into different stages that progressively reduce redundancy. It is closely related to frameworks such as DeepSpeed and is suitable for large-model training and fine-tuning, but it introduces complexity in communication, checkpoints, recovery, and tuning.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Story Mapping

Full ledger: optimizer state, such as Adam's momentum and variance
Every apprentice saves a duplicate: memory redundancy in traditional data parallelism
Sharding: ZeRO shards optimizer states, gradients, and parameters
Running larger models: reducing memory usage
Increased communication: memory savings versus communication cost trade-off
Staged usage: different sharding scopes for ZeRO Stage 1/2/3
Too much state duplication: the zero redundancy goal in the name ZeRO

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

FSDPdata parallelDeepSpeeddistributed training