← Back to Concept Contrast

Concept Contrast Fable

Dual encoders and cross encoders

Conclusion first:One encodes the question and the document separately and then compares distance: fast but limited in accuracy. The other encodes the question and document together: high accuracy but heavy computation. Which you choose determines whether your RAG system is fast and coarse or slow and precise.

One encodes the question and the document separately and then compares distance: fast but limited in accuracy. The other encodes the question and document together: high accuracy but heavy computation. Which you choose determines whether your RAG system is fast and coarse or slow and precise.

Fable

Poetry matching through a wall and poetry appreciation face to face

There was a poetry society in the south of the city that held a poetry judging session once a month. The original method was simple: submitters wrote their poems on paper, and the judges also wrote their preferred style keywords on paper. Staff put the two stacks of paper into two envelopes and sent them to the judging room, ranked them by keyword similarity, and picked the top twenty poems. This process was fast—several hundred poems could be handled in one afternoon.

But then the problem came. One poem contained the line "Bright moon shines among the pines," and keyword matching caught "moon" and "pine," ranking it in the top ten. When the judges looked at it, they found it was a doggerel verse that just happened to use those words. Another poem was exquisite throughout, but the author used rather obscure words, the keywords didn't match, and it was filtered out directly.

The director felt something was wrong. At the next poetry review, he changed his approach: he placed each poem together with each judge's taste description and read them side by side, word by word, checking whether the whole poem's imagery, structure and allusions matched. Accuracy improved greatly; good poems were no longer missed, and poor poems no longer slipped in. But the cost was obvious: what used to be several hundred poems in an afternoon became only a few dozen in an afternoon.

The president held a retrospective meeting and gave the two methods separate names. The first was called 'poetry through the wall,' where the poem and the commentary were processed separately and only compared for distance at the end. The second was called 'face-to-face poetry tasting,' where the poem and commentary were bound together and carefully reviewed.

Later the poetry society worked out a combined approach: for initial screening, use poetry matching through a wall to quickly pull a candidate pool of fifty from several hundred poems. For fine selection, use face-to-face poetry appreciation to carefully rank those fifty and pick the top ten.

After this process ran for a year, the poetry review gathering's reputation rose a notch. The quality of selected poems became consistent, and review time did not spiral out of control. A saying circulated in the poetry society: candidates decided through the wall, rankings decided face to face.

Later someone asked the president why he did not simply use face-to-face poetry evaluation to handle all submissions. The president said: if you had unlimited time and compute, of course you could, but we do not. A good method spends limited resources on the most valuable segment.

Later, the master had the quick-eyed apprentice pick ten first, then had the detail-eyed apprentice compare them one by one. With this division of fast and slow work, the portrait shop could finally both keep up with people and pick accurately.

Bi-Encoder

Encode queries and documents independently into vectors, then match by vector similarity (such as cosine distance). Encoding can be precomputed and cached, making retrieval extremely fast, but it loses fine-grained interaction information between query and document.

Cross-Encoder

Concatenate the query and document and feed them into the model together, letting the model consider the interaction between the two in its attention mechanism. Accuracy is far higher than a dual encoder, but every query-document pair requires a full inference pass, cannot be precomputed, and is computationally expensive.

Story Mapping

  • Poetry across the wall: Bi-Encoder, encoding separately, quickly computing similarity.
  • Face-to-face poetry critique: Cross-Encoder, concatenated input, fine-grained interactive scoring.
  • A doggerel that happens to match keywords: Bi-Encoder semantic drift, where vectors are similar but actually irrelevant.
  • Good poems with obscure words get filtered out: Bi-Encoder has insufficient recall for rare expressions.
  • A combined process of initial screening + fine selection: Bi-Encoder coarse ranking → Cross-Encoder fine ranking, the classic two-stage retrieval strategy for RAG.

In the project

Do not choose one or the other in a RAG system. The standard approach is: Bi-Encoder performs first-stage retrieval, recalling Top-K from a massive document set (usually 50-200); Cross-Encoder reranks the Top-K and outputs Top-N to send to the large model. The cost-effectiveness of this pipeline is far higher than using either one alone. The only thing that needs tuning is the K value—if K is too small, it will miss good documents that the Cross-Encoder could have rescued; if K is too large, the Cross-Encoder's cost will spiral out of control.

京ICP备2026014272号