← Back to concept explainers

Concept Fable

Chunking

Conclusion first:Chunking is the process of splitting long documents into small segments suitable for retrieval and for fitting into a model's context. RAG systems usually cannot stuff an entire document directly into the model; instead, they first chunk the document, vectorize it, index it, and then recall relevant chunks based on the question. Chunking quality directly affects recall precision, context completeness, citation accuracy, and answer quality. Good chunking is not mechanically cutting by word count; it must consider semantic boundaries, heading structure, overlap, chunk size, parent-child chunks, and token budget.

Chunking · Retrieval preparation

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The copyist cut the long scroll into pages that could be found

The archive held a batch of very long files. Some went from city gate repairs to bridge taxes, others from caravan disputes to dock regulations. The curator knew these files were important, but every time someone asked a small question, he had to spread out the entire scroll.

The earliest approach was to store whole volumes. This was the least trouble, and the files were complete. But when a reader asked "can a temporary shelter be put up on the south side of the East Bridge," the retriever could only haul over an entire volume of the "Annual Chronicle of Streets and Bridges," ninety percent of which was irrelevant.

Someone found it too long, so they cut each scroll into fixed lengths. Every ten feet, one section, all neat and tidy. For the first few days, speed increased. Then the problems came: a key rule often had its first half in the previous section and its second half in the next; readers saw only half the meaning.

So the copyist switched to cutting by heading. Each time he came to a subheading, he started a new page. This was better than cutting every ten feet, but some files were written messily, with few headings; some headings looked fine, yet underneath were mixed three completely different problems.

A young scribe began by reading the content first. He discovered that the place that truly needed to be cut was not where the ruler reached, nor necessarily where the heading was, but where the meaning changed boundaries: from bridge structure to taxation, from the original text of regulations to adjudication cases, from background to exception conditions.

He also left a small overlap with adjacent pages. Because some issues straddle the boundary, cutting them off completely would leave the next reader without context.

Later, the director ruled: before a long scroll enters the archive, it must first be cut into pages suitable for questioning and retrieval. Pages too large bring back too much noise; pages too small break the meaning apart; cut the boundaries wrong and the answer is skewed from the very start.

Readers still saw the curator's answer, but what really determined answer quality was often the page-splitting work done before those questions arose. The archive staff then understood: material cannot be used just by putting it in. Long knowledge must first be cut into units that can be retrieved, understood and cited. Later, when the archive reviewed several wrong answers, it found that many errors were not the curator misreading, but pages cut awkwardly. Some people got only the rule, not the exception; others got the case but missed the preceding scope of application. The curator then required page-splitters to write down the reasoning for boundaries, and regularly re-cut pages that were retrieved but useless, and pages that should have been retrieved but weren't. Page-splitting went from odd job to a craft that determines retrieval quality.

Reveal

This story is about: chunking

Chunking is the process of splitting long documents into small segments suitable for retrieval and for fitting into a model's context. RAG systems usually cannot stuff an entire document directly into the model; instead, they first chunk the document, vectorize it, index it, and then recall relevant chunks based on the question. Chunking quality directly affects recall precision, context completeness, citation accuracy, and answer quality. Good chunking is not mechanically cutting by word count; it must consider semantic boundaries, heading structure, overlap, chunk size, parent-child chunks, and token budget.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Very long files: enterprise PDFs, web pages, manuals, contracts, knowledge base articles and other long documents
  • Whole-volume ingestion: no chunking or chunks too large leads to noisy retrieval and high cost
  • One segment every ten feet: fixed-length segmentation, simple but prone to cutting through semantics
  • Split by headings: structured chunking, dependent on the document's own formatting quality
  • Change boundaries by meaning: semantic chunking, keeping the topic within a chunk as complete as possible
  • Adjacent pages leave overlap: chunk overlap, using a small amount of repetition to reduce the risk of boundary breaks
  • The trade-off between pages too large and too small: how chunk size affects recall and context completeness
  • The page-cutting work before problems occur: chunking is upstream engineering for RAG, not a remedy applied only at the answering stage

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

document parsingtoken budgetoverlapsemantic chunkingretrieval granularitycontext window
京ICP备2026014272号