← Back to concept explainers

Concept Fable

Answer Relevance

Conclusion first:Answer Relevance measures whether a model's answer truly responds to the user's question and task objectives. An answer can be fluent, truthful, and informative yet still irrelevant, because it does not address what the user actually asked. For RAG and Agent systems, answer relevance is an important metric for evaluating output quality, used to prevent the system from giving answers that seem reasonable but deviate from the task.

Answer Relevance · RAG evaluation

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The person asking for directions receives only a poem praising the street

A stranger came to Willow Bridge Town for the first time and wanted to find the clinic. He stopped a very talkative guide and asked: How do I get to the clinic?

The guide immediately began introducing the streets of Liuqiao Town, saying East Street has a long history, West Street has bustling shops, South Bridge has the best scenery, and North Gate has been repaired three times. He spoke fluently and politely, with no obvious errors.

After hearing this, the outsider still stood in place, not knowing which way to go. The guide did not think he had answered wrongly: everything I said is true information, and all of it relates to roads. The outsider pressed, "Then which way do I go now?" The guide added another passage on the history of city defenses, every sentence supported, yet still leaving the person standing there.

An old man selling tea nearby took out a small piece of paper and wrote a few lines: from here, walk forward two streets, turn right at the stone lion, pass the pharmacy, and after fifty more steps you'll reach the clinic. The stranger followed the directions and arrived quickly.

The mayor later asked the guide and the old woman to review the matter. The guide had more information, and the old woman spoke more briefly. But for that question, the old woman was more useful, because every sentence she spoke advanced the outsider's goal. The old woman did not always speak less. If the outsider asked whether the clinic was good at bone injuries, she would switch to doctors, hours and consultation fees; her standard was not brevity, but staying close to the question.

The guide was still unconvinced. The next day, the question was changed to "Why is the medical clinic on North Street?" This time, his knowledge of street history was useful. The mayor said that the same piece of knowledge, placed in the wrong question, is just a detour.

From then on, the town's guides no longer competed only over who knew more or who spoke more elegantly. They began to ask: Does this sentence directly answer the question? Does it help the other person move one step toward their goal? Does it pull attention off track? The mayor had the guides practice restating the goal first, then cutting away the branches that couldn't help anyone act. Less information remained, but the answers became more reliable. The guides later divided each response into two columns: the route that must be answered, and background that might be useful. When a guest is in a hurry, give the route first; when a guest asks idly about the town's history, then expand. Relevance changes with the question, not with how much the speaker knows.

Liuqiao Town learned that relevant does not mean touching on the topic; relevant means being useful to the problem at hand.

Reveal

This story is about: answer relevance

Answer Relevance measures whether a model's answer truly responds to the user's question and task objectives. An answer can be fluent, truthful, and informative yet still irrelevant, because it does not address what the user actually asked. For RAG and Agent systems, answer relevance is an important metric for evaluating output quality, used to prevent the system from giving answers that seem reasonable but deviate from the task.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • A stranger asks how to get to the clinic: an explicit task or question raised by the user
  • The guide talks about street history: the content is related to the topic, but it does not answer the user's real need
  • True but useless information: factual correctness does not equal answer relevance
  • The tea seller gives directions: an answer that directly serves the user's intent
  • Every sentence sticks to the goal: the manifestation of high answer relevance
  • Does it advance the asker's goal: the core question for evaluating answer relevance
  • Relevance is not tangency: semantic similarity does not equal task relevance

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

relevancesemantic similarity

Related distinctions

This concept is easily confused with adjacent concepts; it is recommended to continue to the comparison page.

京ICP备2026014272号