Concept Contrast Fable

Retrieval-augmented generation and knowledge base question answering

Conclusion first:Many people build a knowledge base, hook up a large model, and say they've done RAG. In fact, retrieval-augmented generation is a complete systems engineering effort—including data chunking, vectorization, retrieval, ranking, generation, and evaluation; knowledge-base Q&A is just one application scenario. Equating the whole engineering effort with a single feature interface most often leads to this: it retrieves but answers wrong, then everyone goes back to fix the knowledge base, and it's still wrong after the fix.

Many people build a knowledge base, hook up a large model, and say they've done RAG. In fact, retrieval-augmented generation is a complete systems engineering effort—including data chunking, vectorization, retrieval, ranking, generation, and evaluation; knowledge-base Q&A is just one application scenario. Equating the whole engineering effort with a single feature interface most often leads to this: it retrieves but answers wrong, then everyone goes back to fix the knowledge base, and it's still wrong after the fix.

Fable

The archivist of the old library and the archive lookup desk at the counter

The old library held three hundred years of archives and employed an archivist, Old Zhou. Old Zhou's job wasn't just finding books on the shelves—he first had to split each year's new archives into files of different sizes, tag each file, then build retrieval cards by tag, and then decide how long a summary to write on which spines. To outsiders, he did just two things: take query slips and hand over archives.

Later, the archive set up a lookup desk at the main entrance, letting visitors enter questions themselves. The machine searched the index card library, found relevant archive excerpts, pieced together a reply, and handed it to the visitor. The visitors were delighted, feeling that at last they no longer had to wait for Old Zhou.

The problem emerged in the second month. A scholar came to audit the province's salt and iron tax accounts, and the archive desk told him that the tax amount for a certain year was thirty thousand taels. The scholar excerpted it and published it, and a journal reviewer pointed out that the figure was completely wrong. The scholar stormed back to the archive, and Old Zhou pulled out the same batch of index cards, pointing to a note: the archive desk had drawn the seasonal estimate, while the actual receipts were in the next volume, but it did not understand the continuity between volumes and cited it with full confidence.

The archive therefore summoned Old Zhou and the engineers from the records desk to sit face to face for a whole day. Only then did the engineers realize that they had previously only changed the labeling system of the retrieval card catalog, without checking the retrieval ranking rules—the records desk always ranked the archive with the most similar title first, and never asked whether that archive was the authoritative version or had errata.

Old Zhou drew a diagram: from the original record to the index card to the ranking rules to the excerpting logic to the final reply, every step could go wrong. The lookup desk was only the topmost interface of this diagram, and fixing it required changing several layers underneath—the splitting method, the labeling strategy, the ranking weights, the sourcing annotation habits. None of these could be solved by adding a few more books to the knowledge base.

Finally the archive set a new rule: from then on, when someone reported an inaccurate answer, first write a dispute marker next to the retrieval card, and do not rush to dismantle the archive. Only after ten consecutive disputes pointed to the same tagging strategy would that tag be redone. Old Zhou said this treats the root cause, not the symptom.

When the scholar came again later, he found that after each citation on the archive desk, the source archive number and whether it was an authoritative version were noted, and he no longer needed to go back to Old Zhou to verify. A line had been added at the entrance to the stacks: between finding it and stating it correctly lies an entire engineering effort.

At year-end, the archive moved the cost of the reference desk from the procurement budget to the operations budget—because its real cost was not in that machine, but in the people continuously optimizing the retrieval pipeline.

Retrieval-Augmented Generation

A complete systems engineering effort covering data preprocessing, index construction, vector retrieval, result ranking, context assembly and large model generation, where every step affects final answer quality.

Knowledge base Q&A

A specific application scenario of RAG, manifested as users directly doing natural language Q&A on a knowledge base; it depends on the underlying RAG engineering pipeline, but itself does not include governance of indexing and retrieval strategies.

Story Mapping

Old Zhou's seven steps as an archivist (splitting, tagging, retrieval cards, sorting, excerpting): the complete engineering pipeline of RAG
Front desk archive station: the application interface for knowledge base Q&A, representing only the topmost layer
Misquoted salt and iron tax ledgers: a retrieval ranking problem mistaken for an inaccurate knowledge base
New rule for dispute flags: don't tear down the whole archive over one error; trace it back to a specific link in the chain
Source attribution and authoritative versions: the importance of ranking and traceability in the RAG pipeline

Applying it to Andy Ren Xu's project

When a client says the knowledge base is inaccurate, do not immediately add documents. First look at what retrieval returned, whether ranking pushed noise to the front, and whether chunking cut answers into fragments. Fixing the three-layer pipeline is more effective than adding ten books. When quoting a client, also separate the work: building a knowledge-base Q&A interface is a small project, while connecting the entire RAG pipeline and continuously optimizing it is a much larger investment.

Retrieval-Augmented Generation Knowledge base Q&A