Concept Fable

Linear attention

Conclusion first:Linear Attention is a class of methods that reduce the computational complexity of attention from the quadratic growth of traditional self-attention to approximately linear growth. It usually avoids direct pairwise comparison of every token with every token through kernelization, feature mapping, or state summarization, thereby improving the cost and latency of long-sequence inference. Enterprises should evaluate the quality loss caused by approximation when using it, especially in tasks requiring fine alignment and complex reasoning.

Linear Attention · Architecture

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The marketplace no longer has everyone shake hands with everyone else

There was an old rule at the market meetings: before each vendor spoke, they had to shake hands with every other vendor to confirm what each cared about. With few people it was lively; with many, meetings dragged from dawn to late night.

The new market expanded to several hundred stalls, and the handshake rule became a disaster. Vegetable vendors had to confirm one by one with cloth merchants, blacksmiths, and tea merchants; by the time it was their turn to actually negotiate, it was already dark. The guild leader first shortened the speeches, but the problem remained—the number of handshakes skyrocketed with the number of people.

Someone suggested having the stallholder shake hands only with the ten people nearby. It was faster, but it missed genuinely relevant transactions farther away, such as the salt merchant and the pickle shop. The first fix saved time but also lost important connections.

Later the guild leader changed the method. Each stall owner first wrote their needs as several public tags and put them into a basket in the center of the square; when speaking, they no longer shook hands one by one, but read these aggregated tags and then, combined with their own goods, decided whom to pay attention to.

This approach no longer preserves the complete relationship between every pair of people, so there is a trade-off in granularity. But for many long meetings, it lets discussion time grow steadily with the number of people, no longer dragged down by pairwise handshakes. Key distant relationships can also surface through public markers.

The market finally closed before dusk. The guild leader understood that as participants increased, it was no longer possible to always have everyone compare directly with everyone else; replacing pairwise comparison with an accumulable intermediate representation could bring long-sequence processing back from explosive cost to an affordable range.

The guild master later discovered that public tags also need good design. If tags are too coarse, every request blurs into "wants to make money"; if too fine, you are back to everyone confirming one by one. The right representation determines whether this approach can be both fast and useful.

The market kept a few direct meetings. Major contracts and rare disputes still had both parties sit down for detailed talks; routine coordination went through the central basket. In this way, both near and far relationships had their place, and the cost of long meetings no longer multiplied with the number of stalls. Later, the guild leader taught newcomers to look at the meeting ledger: if the number of people doubles, the time quadruples, and the old rules will eventually fail. Public markers are not perfect understanding, but they let a large market continue to collaborate. It sacrifices a bit of pairwise detail in exchange for long meetings that can still be held. The guild leader treated this as a survival condition of the grand market, not just a small trick to save time.

Reveal

This story is about: linear attention

Linear Attention is a class of methods that reduce the computational complexity of attention from the quadratic growth of traditional self-attention to approximately linear growth. It usually avoids direct pairwise comparison of every token with every token through kernelization, feature mapping, or state summarization, thereby improving the cost and latency of long-sequence inference. Enterprises should evaluate the quality loss caused by approximation when using it, especially in tasks requiring fine alignment and complex reasoning.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

Every stallholder shakes hands with every other: the pairwise token interactions of standard self-attention
Meeting explosion as the number of people grows: O(n²) cost rises with sequence length
Public ledger board: the aggregation or feature mapping mechanism in linear attention
Logistics scheduling long sequences: low-latency long-context scenarios
Public ledger boards are approximate: efficiency gains may sacrifice fine-grained interaction
Key steps retain standard attention: architectures can be layered by task
Redesigning information aggregation: the core motivation of linear attention

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

SSMefficient attentionlong context