← Back to concept explainers

Concept Fable

Prompt caching

Conclusion first:Prompt Caching reuses previously processed prompt content or prompt prefixes so repeated instructions, policies, examples, and reference material do not have to be fully processed on every request. For enterprise LLM applications, it can lower latency and cost when requests share stable prompts, documents, or workflow scaffolding. It differs from response caching: the model still generates a fresh answer for the variable part, but avoids recomputing reusable context.

Prompt Caching · Inference optimization

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The newspaper lays out its regular pages in advance so the news can make the morning edition

The city newspaper was published every morning. Each edition had fixed columns, byline rules, prohibited words, and advertising slots, yet the editors started typesetting from blank lead plates every night.

As the news grew, the reporters had already delivered their drafts, but the print room was still stuck on a fixed layout. Readers complained the newspaper arrived late, and the curator assumed the reporters were writing slowly.

He first pressed the reporter to submit the article early, then made the typesetter work overtime. As a result, the news body did arrive early, but the type was still being set again and again for the same masthead, footer, and statement.

Someone pasted commonly used wording on the wall, but every time they set type they still had to reconfirm: is this the same version of the rules, has the advertisement changed, is a certain statement still valid. The prompts on the wall helped people, but did not help the typesetting desk.

A supplement editor split the newspaper into two layers: stable, unchanging layout rules and the news that changes every night. The stable part was laid out first as a base plate, and the changing part was embedded into it; once the base plate was revised, it was marked with a new number.

She also stipulated that advertising plates for different customers could not be shared, and that once legal statements changed, old base plates had to be discarded. Similar did not mean identical, and speed could not override boundaries. The first time someone misused an old base plate, she preferred to re-layout rather than let the wrong version enter the printing press.

The newspaper office's morning pace changed quickly. Editors no longer spent time repeatedly confirming fixed requirements, but focused on checking news facts, headline risks, and the paragraphs readers cared about most.

The curator then realized that much of the delay did not come from new content, but from repeatedly processing the same batch of instructions and background materials. Keeping the stable parts ready in advance gave the later work room to spare; but what was kept ready also had to know when it would expire. Later, whenever a new special topic was started, the supplement editor first asked three things: which layout rules were stable, which content changed daily, and which changes would invalidate the old base plates. The newspaper thus had much less repetitive work, and fewer misprints caused by rushing. Speed came from reuse, reliability came from expiration rules. The printing room later had far fewer late-night arguments. Everyone knew which lead plates could be used directly and which had to be reset; speed no longer depended on staying up late, but on stable parts being processed in advance. Later, during the retrospective, everyone wrote this lesson into the daily rules: first look at where on site the result truly changed, then decide what to keep, what to repair, and what to restrict.

Reveal

This story is about: prompt caching

Prompt Caching reuses previously processed prompt content or prompt prefixes so repeated instructions, policies, examples, and reference material do not have to be fully processed on every request. For enterprise LLM applications, it can lower latency and cost when requests share stable prompts, documents, or workflow scaffolding. It differs from response caching: the model still generates a fresh answer for the variable part, but avoids recomputing reusable context.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Fixed newspaper house rules: system prompt, workflow instructions, few-shot examples, and shared reference materials
  • Starting from a blank version every night: each request reprocesses the full prompt
  • Base layer: reusable context in prompt cache or prefix cache
  • Version number: prompt version and cache invalidation
  • Customer ads cannot be shared: tenant isolation and permission isolation
  • Morning speed: lower TTFT, lower prefill cost
  • Final insight: Prompt Caching caches the processing results of stable context, not fixing all answers in place

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

KV cacheprefix cachinginference costcontext windowsystem promptlatency optimization
京ICP备2026014272号