← Back to concept explainers

Concept Fable

Residual connections

Conclusion first:Residual Connection adds a layer's input directly to its output, letting the network learn incremental changes rather than completely rewriting the representation at each layer. It improves gradient propagation and training stability in deep networks, and is a key reason Transformers can stack many layers. On the enterprise agent side, this is felt as stronger model capability and more stable training; in model compression, fine-tuning, and architecture selection, residual structures also affect trainability.

Residual Connection · Architecture

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

An elevated bridge that preserves the original route at every level

A mountain city built a multi-story workshop, and materials were reprocessed on every floor. At first everyone was excited, thinking the more floors, the finer the product.

Soon the problem appeared. After the material passed through more than a dozen layers, the original key information had been altered away; if one layer made a small mistake, later layers could only keep processing on top of the error. The apprentices only stared at the final product and did not notice that the material in the higher layers increasingly resembled dough that had been kneaded over and over, with its original grain and dimensions hard to recognize.

The foreman first reduced the number of layers. Errors decreased, but capability weakened too, and complex orders could no longer be handled.

The second fix was to have each layer write detailed explanations. The explanations piled up beside it, but could not restore material that had already been altered. The explanatory notes were also often ignored by later layers. The later masters faced already deformed material, and even if they knew what it originally looked like, it was hard to restore it based on a note.

Later, engineers built an elevated bridge for every layer. Materials could be processed at that layer, or the state before entering that layer could be sent directly to the exit and then combined with the processing result.

This original path meant the deep workshop no longer feared continuing to build higher. If later layers could not learn useful changes for the time being, they could at least preserve the earlier information; and if a layer needed only a small change, it did not have to rebuild everything. When the overpass was first built, some worried it would make the current layer lazy. The engineers stipulated that the bridge was not a way to bypass processing, but to let processing bear only valuable changes.

A few months later, the workshop could take on more complex work. The foreman found that depth was not the problem; layer upon layer of processing with no return path was the problem. The workshop later grew to dozens of layers, yet still regularly checked whether the elevated bridge was clear. If the bridge was blocked, deep layers would start forgetting again; if the bridge was too wide, each layer might learn nothing. Retention and change had to be balanced together. This bridge also made new apprentices bolder about adding layers. In the past, every added layer felt like a gamble, fearing the earlier craft would be washed away; now each layer only needed to learn the difference it should fill, and the workshop could go deep while still tracing back where the change truly came from.

He finally said that a complex workshop must allow new processing while also preserving a way back. Each layer can change a little on top of the original foundation, so depth does not become amnesia.

Reveal

This story is about: residual connections

Residual Connection adds a layer's input directly to its output, letting the network learn incremental changes rather than completely rewriting the representation at each layer. It improves gradient propagation and training stability in deep networks, and is a key reason Transformers can stack many layers. On the enterprise agent side, this is felt as stronger model capability and more stable training; in model compression, fine-tuning, and architecture selection, residual structures also affect trainability.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Multi-layer workshop: deep neural network
  • Key information altered away: information and gradient decay in deep networks
  • Reduce layers: a patch that sacrifices capability
  • Overpass: Residual Connection
  • Add the processed result to the original state: residual addition
  • The deep workshop keeps building higher: training deeper models becomes possible
  • Final insight: deep capability requires preserving information pathways

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

layer normalizationtransformer blockvanishing gradients
京ICP备2026014272号