← Back to concept explainers

Concept Fable

Auxiliary loss

Conclusion first:Auxiliary Loss / Load Balancing Loss is an additional loss term in MoE training, used to encourage tokens to be distributed more evenly among experts, avoiding routing collapse or overloading of a few experts. It does not replace the main task loss, but adds a system health constraint beyond the quality objective, making expert utilization, training stability, and throughput more controllable.

Auxiliary Loss / Load Balancing Loss · MoE

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The line in front of the star craftsman finally dispersed

In the south of the city there were ten specialized craftsmen handling different tasks: fixing locks, mending shoes, carving seals, and filling prescriptions. The apprentice who assigned work wanted to save trouble, so he always gave orders to the two he knew best, a long line formed at the door, and the other craftsmen sat idle.

The shopkeeper looked only at total daily revenue and at first did not notice a problem. It was not until popular masters grew tired and made mistakes, less popular masters grew rusty, and customers complained about long waits. The total account still looked acceptable, but internally it was already out of balance.

The new manager added a small account item for the apprentice assigning work: besides checking whether the work was completed, he also checked whether orders were excessively squeezed toward a few masters. If some were worked to exhaustion while others were idle for a long time, the work assignment would be penalized.

At first the apprentice assigned work randomly in the name of evenness, sending a medicine list to the seal carver. The steward reminded him again that balance cannot override fit; first make sure work goes to the right person, then avoid crushing a few under the load.

The shopkeeper first ordered the apprentice to distribute work evenly. As a result, the cobbler received a medicine prescription, and the seal carver was forced to fix locks, making customers even more dissatisfied. Even distribution is not the goal; work still has to go to the right people, but a few must not be crushed.

Later, when assigning orders, the shopkeeper added a small account: if a craftsman was already scheduled too full, new orders would first go to the second most suitable person; if a craft had long been idle, suitable work was also arranged to keep it proficient. Total revenue still had to be considered, and internal capacity also had to be considered. A shop does not just need to sell today; it also needs every counter to be able to work over the long term.

Later, before assigning work, the apprentice would look at two boards at once: who is best suited to this order, and who in the shop is already overloaded. Looking at both boards together prevents sacrificing tomorrow's capable workers for today's fastest result. Later, before assigning work, the apprentice would look at two boards at once: who is best suited to this order, and who in the shop is already overloaded. Looking at both boards together prevents sacrificing tomorrow's capable workers for today's fastest result. Later, before assigning work, the apprentice would look at two boards at once: who is best suited to this order, and who in the shop is already overloaded. Looking at both boards together prevents sacrificing tomorrow's capable workers for today's fastest result.

A few weeks later, the shop was more stable. The main ledger measured whether things were done well, and the small ledger reminded whether the division of labor was balanced. Without this small ledger, surface success would mask internal congestion.

Reveal

This story is about: auxiliary loss

Auxiliary Loss / Load Balancing Loss is an additional loss term in MoE training, used to encourage tokens to be distributed more evenly among experts, avoiding routing collapse or overloading of a few experts. It does not replace the main task loss, but adds a system health constraint beyond the quality objective, making expert utilization, training stability, and throughput more controllable.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • A long line at the swordsmith's door: a few experts are over-selected
  • Other masters idle: expert under-utilization
  • Forced round-robin assignment: pursuing only equality sacrifices task-match quality
  • Additional nighttime penalty: auxiliary loss / load balancing loss
  • Still matched by task: the main task loss still drives quality
  • Avoid long-term overload and learning loss: maintain expert utilization and training stability
  • Final insight: load-balancing loss makes MoE not only learn to answer questions, but also learn to use expert capacity healthily

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

MoEexpert routingload imbalance
京ICP备2026014272号