← Back to concept explainers

Concept Fable

Feedforward network

Conclusion first:The Feed-Forward Network (FFN) is the position-wise feed-forward network located after attention in a Transformer layer, usually composed of two linear transformations and a nonlinear activation. Attention is responsible for exchanging information between tokens, while the FFN is responsible for feature transformation and expansion-compression of each token's representation. In enterprise Agents, the FFN is not directly visible, but it carries the model's internal knowledge and pattern transformations, and also affects model size, inference cost, and quantization compression strategies.

Feed-Forward Network, FFN · Architecture

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Small workshops in every cell

The sorting house in the city processed letters from everywhere each day. Upstairs there was a row of comparison desks, where the clerks first compared each letter's relationship to the others: which letters belonged to the same case, which supplemented which, and which should be prioritized.

The mutual-check desk was very useful, but it could not actually finish processing the letters. After comparison, each letter still needed its own marks: whether it was urgent, whether a receipt was required, whether it contained silver notes, whether it should go to a government office. The new clerk mistakenly thought that once the mutual comparison was done, sorting was complete.

When the first error occurred, two similar letters were both sent to the ordinary slot, and one of them actually contained a deadline. The supervisor first had the clerk discuss several more rounds at the mutual review table, but the more they discussed upstairs, the slower it got, and the judgment that should have been made for a single letter was still not completed by anyone.

The old sorter sends each letter into a row of small workshops. Each workshop sees only the clues the letter already carries, first expanding its observations, then compressing them back into a few clearly marked labels. It does not talk to other letters, yet it makes each letter's own meaning clearer.

The small workshop was looked down on at first, like odd jobs behind the mutual observation desk. But without it, no matter how clear the relationships were, they could not land on the handling decision for a single letter; with it, letters could turn the clues found through comparison into their own next state.

The sorting house later developed a rhythm: first look at the relationships between letters, then let each letter pass through the processing table on its own. The supervisor understood that group relationships and individual transformations are both indispensable; the latter is quiet but is a core step in turning understanding into a usable representation.

Later, as the letters grew longer, the supervisor discovered that the small workshop also handled another task: reorganizing the clues obtained from earlier relationship comparisons into a form the next layer could continue to use. If this step was rough, later judgments would carry fuzzy markers and continue to go wrong.

The new clerk finally stopped treating the small workshop as miscellaneous chores. The mutual-viewing table let letters know each other, and the small workshop let each letter update itself. The two processes alternated, so the sorting building could both see global relationships and preserve the operable state of individual letters. Later, when the supervisor checked sorting quality, they looked at whether the mutual-viewing table found relationships and whether the small workshop turned relationships into clear labels. The former kept letters from being isolated; the latter kept letters flowing. Without this local processing, the earlier relationships would be hard to turn into later capabilities.

Reveal

This story is about: feedforward networks

The Feed-Forward Network (FFN) is the position-wise feed-forward network located after attention in a Transformer layer, usually composed of two linear transformations and a nonlinear activation. Attention is responsible for exchanging information between tokens, while the FFN is responsible for feature transformation and expansion-compression of each token's representation. In enterprise Agents, the FFN is not directly visible, but it carries the model's internal knowledge and pattern transformations, and also affects model size, inference cost, and quantization compression strategies.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Lamp array: Attention mechanism
  • Each cell: a token representation in the sequence
  • Small workshop: FFN
  • Shine longer: a patch that only expands attention computation
  • Handwritten judgment: stacked external rules
  • Exchange first, then process: how Attention and FFN work together in a Transformer layer
  • Final insight: model capability comes from the combined effect of relational modeling and position-wise transformation

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

activation functiontransformer blockSwiGLU
京ICP备2026014272号