← Back to concept explainers

Concept Fable

Tensor parallelism

Conclusion first:Tensor parallelism splits a single weight matrix or tensor operation across multiple GPUs. Each device computes part of the same layer, and the results are then combined through communication.

Tensor Parallelism · Distributed inference

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The giant millstone was assigned to four waterwheels turning together

In the valley there was a giant mill, its millstone so large that a horse could not turn it. In early years the mill only ground wheat, and a small millstone was enough, with a simple water channel.

Later, the city needed to grind a kind of hard grain, which required a giant millstone. A single waterwheel could not drive it, and if the millstone stayed in one place, the whole mill would stop work. The mill owner first blamed the horse for not being strong enough, then changed to a thicker rope. The rope broke twice, the waterwheel house was pulled crooked, and the giant mill still moved only half a turn.

The workshop owner first divided the grain into four piles and gave them to four small mills to process separately. But hard grain required the same large millstone pattern, and the flour from the small mills came out with inconsistent coarseness, unusable together.

He also had four water wheels take turns pulling the giant mill. Each water wheel grew tired, yet the millstone stopped and started, and the flour path was sometimes coarse and sometimes fine. When taking turns pulling, each water wheel thought it should exert force only on its turn; just as one stretch of force receded, the next had not yet connected, and the grain was crushed in the gap.

The old mechanic divided the giant millstone's gear ring into four segments, with each waterwheel responsible for one segment of power. The four segments had to mesh at the same time, rotate in sync, and finally combine into a complete grind in the same circle.

This required precise canal signals. If any waterwheel was half a beat slow, the whole set would shake. For this, the mill added a synchronization rod and an echo bell to ensure the four forces aligned at the same moment. The synchronization rod was initially poorly tuned, and the four gear rings fought each other for power. The old mechanic had each waterwheel idle in sync first, then feed in a small amount of grain, confirming the vibration fell within a controllable range.

The giant mill finally turned. No single waterwheel owned the entire millstone, but each waterwheel calculated its own portion, and together they formed the complete work. The mill owner also learned to calculate the cost of splitting. Split too little, and a single waterwheel still struggled; split too finely, and the synchronization rods and echo bells would eat up the gains. Before cutting up a big job, you must first know the cost of putting it together. Later, the giant mill took on nighttime orders, and the water channel pressure varied even more. The mill stationed watchers beside each gear ring, and when they found which section was overheating or which was off-beat, they adjusted the water immediately. A big job completed jointly requires both splitting and continuous coordination.

The workshop owner understood: a task that is too large does not have to be crammed into a single mechanism; you can split the same layer of force and let multiple mechanisms complete it together.

Reveal

This story is about: Tensor Parallelism

Tensor parallelism splits a single weight matrix or tensor operation across multiple GPUs. Each device computes part of the same layer, and the results are then combined through communication.

It enables ultra-large models that can't fit or run on a single GPU to run across devices, but it introduces synchronization and communication overhead. The larger the model and the more devices, the more critical the communication design.

Metaphor mapping

  • The giant millstone: enormous weight matrix or tensor operations within a single layer
  • Four waterwheels: multiple GPUs
  • The gear ring is divided into four segments: splitting the same layer of computation across different devices
  • Sync rods and echo bells: cross-GPU communication and synchronization
  • Four small mills each grind their own failure: simple data splitting cannot solve the problem of a model layer that is too large
  • Full crush: merging partial results to get the layer output

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

pipeline parallelismmodel parallelismdistributed inferenceMegatron
京ICP备2026014272号