Concept Contrast Fable

Evaluation and LLM evaluation: one is the ruler, the other is using that ruler to measure a specific model

Conclusion first:Evaluation is a general action—designing metrics, building benchmark datasets, calculating scores. LLM evaluation is the specialized application of this action to large language models, requiring additional handling of open-ended outputs, subjective judgments, and prompt variations. Applying general evaluation methods directly to LLMs produces good-looking scores that don't match real performance.

Evaluation is a general action—designing metrics, building benchmark datasets, calculating scores. LLM evaluation is the specialized application of this action to large language models, requiring additional handling of open-ended outputs, subjective judgments, and prompt variations. Applying general evaluation methods directly to LLMs produces good-looking scores that don't match real performance.

Fable

There are two rulers: one measures all cloth, the other measures only indigo cloth

In the dye works there was an old cloth-measuring master with a brass ruler. He used the same ruler for every fabric—wide, narrow, cotton-linen, silk—and one pass told him the size. The dye works had more and more fabric varieties, but the old cloth-measuring brass ruler never changed, and everyone who looked at cloth said the sizes were accurate.

Later, the dye house brought in a new kind of indigo cloth—it shrank after absorbing water and faded after long exposure to light. The dimensions measured with the same copper ruler were accurate, but the actual wearing dimensions after getting wet did not match at all. A batch was returned to the warehouse because the cloth met the measurement standard but failed the color standard. The buyer pointed at the quality inspection report and said: the ruler measured correctly, but you measured the wrong dimension. The real key for indigo cloth was comparing color difference with another batch dyed in the same vat, not comparing dimensions with dry cloth.

Old Master Liang the cloth measurer and his apprentice made a new table. Beside “copper ruler measures dimensions,” they added new columns: evaluation of indigo cloth needed to look at three dimensions—wet-state dimensional change rate, color difference grade within the same vat, and sunlight fading gradient. After each batch of indigo cloth came out of the vat, the apprentice scored it according to the new table. The scoring samples of standard cloth accumulated to forty pieces, each kept in duplicate: one pasted into the archive slot, one left for the apprentice to cross-check.

The dyehouse took on a bigger order—the client sent 120 of their own fabric samples and required them all to be evaluated within one day. The apprentice measured them one by one the old way and was only halfway done by the afternoon. The old fabric-measuring master brought out a new plan: first take four typical fabric samples, ask the accountant to score each item using a unified description template, then write the scoring criteria into a step sheet, and have the workers by the dye vats score and summarize each vat. This system was about how quickly and consistently the dyehouse could produce evaluation results, no longer focusing only on the quality of a single vat of fabric.

At the year-end review, the apprentice brought two reports: one was the annual evaluation report for general cloth—dimensions measured by the brass ruler, cotton and linen by feel, silk by luster, with stable metrics over five years; the other was the dedicated evaluation report for indigo cloth—wet shrinkage better than last year, color variation within the same vat down by half a grade, and the fading of the new dye vat still lacking half a year of data. The two reports differed completely in format, thresholds, and statistical criteria; putting them in the same table would make them unreadable.

The old cloth measurer later wrote a rule on the door panel: evaluation is a general system — you can score any object with any metric. But indigo cloth assessment is just one specialized evaluation; its object dictates that you look at wet state, color deviation, and fading, not just dimensions. After the dye works accepted this rule, every new variety asked one question before entering the vat: which ruler should this batch use?

Evaluation

A general measurement action—designing metric systems, constructing benchmark datasets, and calculating scores—applicable to performance measurement of any system or algorithm.

LLM evaluation

The specialized application of evaluation to large language models requires handling open-ended outputs, subjective judgments, prompt sensitivity, evaluator bias, and other issues. Evaluation frameworks such as LLM-as-judge are commonly used means.

Story Mapping

A bronze ruler measuring all cloth: a general evaluation framework, suitable for measuring systems with standard input-output
Indigo cloth shrinks when wet, and you rely on color difference rather than dry-state dimensions: LLM evaluation needs dedicated dimensions, and generic dimensions may fail to measure the real problem
The new table adds wet state, color differences within the same dye vat, and fading gradients: the metric system for LLM evaluation must cover multiple aspects of quality in open-ended outputs
Forty standard scoring samples cross-validated: LLM evaluation needs reliable scoring benchmarks and cross-validation mechanisms
The accountant scores with a unified description template: evaluation infrastructure, ensuring consistency and reproducibility in large-scale assessment
The two annual reports have completely different formats: general evaluations and LLM-specific assessments cannot be mixed in the same table; the applicable definitions and criteria must align with their respective objects

Building an evaluation system for AI systems

When building an AI evaluation system, first distinguish whether you are building general evaluation infrastructure or a specialized evaluation for a particular model. General infrastructure solves the efficiency problem: how to run scoring quickly, consistently and automatically. LLM evaluation solves the accuracy problem: whether the metrics you measure correspond to the quality users perceive. LLM evaluation requires a great deal of effort on benchmark construction—extracting typical cases from real scenarios, establishing rater consistency calibration, and periodically reviewing the effectiveness of the evaluation framework itself. If a team uses a single classification accuracy or F1 score as the only LLM evaluation metric, it is probably measuring something unrelated to the real user experience.

Evaluation LLM evaluation