← Back to concept explainers

Concept Fable

Cache

Conclusion first:Caching in LLM applications means reusing existing responses, embeddings, retrieval results, tool results, or prompt prefixes to reduce latency, cost, and repeated model calls. Cache design must focus on hit rate, invalidation strategy, semantic equivalence judgment, permission isolation, data freshness, and the risk of caching errors.

Caching · LLM infrastructure optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The teahouse no longer boils the same pot of water from scratch every time

The mountain-road teahouse received travelers every day. Many asked the same things: how many li to the east ferry, where one could change horses, which road would collapse in the rain. Every time a waiter was asked, he ran to the back yard to leaf through the road register, while the stove burned and guests queued.

The shopkeeper first had the assistant answer faster. Faster it was, but someone used yesterday’s sunny road conditions to answer today’s heavy rain, and two trucks got stuck in the mud. If convenience ignores whether information is old or new, it will also save the wrong answers.

Later the shopkeeper placed a small wooden box at the counter, writing the most frequently asked, short-term stable answers on plaques: distance to the east ferry, location of the stables, city gate hours. Behind each plaque was noted when it was written and when to recheck. When rain, landslides, or official road closures occurred, the plaques were immediately flipped over and voided.

The clerk also learned to distinguish: for 'how many li to the east ferry', he could take the answer directly from the box; for 'which route is safest to take porcelain today', he had to consult the ledger and ask the porters, and could not be lazy. Only repeated and equivalent questions deserved an answer taken from the box.

The teahouse first posted common questions and answers on the wall. On sunny days it was fine, but after rain it was a mess; the ferry was open in the morning, but stopped when the water rose in the afternoon. If the assistant only memorized the old answers on the wall, the more skilled they became, the more dangerous it was.

The old shopkeeper switched to two layers of signs: stable matters were written on wooden boards, while things that change—road conditions, ship schedules, horse prices—were written on erasable wet boards, with the last checked time marked. When a guest asked the same question again, the assistant first checked whether it was still fresh; if fresh, answer quickly; if expired, go back to the back room to recheck. The premise of saving time is knowing when the answer will go bad.

The servers also learned which replies must never be given lazily. Directions could be answered from the old board, but a cave-in had to be checked against the new report; tea prices could be answered directly, but whether the apothecary was open had to be confirmed again. Between a fast answer and an accurate answer, there must be a judgment of old versus new. The servers also learned which replies must never be given lazily. Directions could be answered from the old board, but a cave-in had to be checked against the new report; tea prices could be answered directly, but whether the apothecary was open had to be confirmed again. Between a fast answer and an accurate answer, there must be a judgment of old versus new.

The teahouse soon quieted down, leaving firewood and manpower for matters that truly required judgment. Remembering reusable answers is not laziness; it removes the cost of repetition. But if boundaries are unclear, old answers become new trouble.

Reveal

This story is about: caching

Caching in LLM applications means reusing existing responses, embeddings, retrieval results, tool results, or prompt prefixes to reduce latency, cost, and repeated model calls. Cache design must focus on hit rate, invalidation strategy, semantic equivalence judgment, permission isolation, data freshness, and the risk of caching errors.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Teahouse: LLM / Agent application service
  • Boiling water anew each time: every request fully calls the model, retrieval and tools
  • The common answer on the wall: coarse static caching
  • Several small cabinets: response cache, retrieval cache, embedding cache, prefix cache and other cache layers
  • Deadlines and applicable conditions: TTL, invalidation, scope and cache policy
  • Similar-meaning questions: semantic caching
  • Firewood saved: token cost and latency savings
  • Final insight: caching reduces cost and latency through controlled reuse, but must avoid reusing expired, unauthorized or incorrect results

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

semantic cacheprompt cacheresponse cacheembeddings cachegatewaycost optimization
京ICP备2026014272号