Concept Contrast Fable

Alignment evaluation and model safety evaluation

Conclusion first:Alignment evaluation checks whether the model is obedient, understands intent, and acts according to human preferences. Safety evaluation checks whether the model produces harmful content and whether it can be induced to do dangerous things. An obedient model is not necessarily safe, and a safe model does not necessarily obey you.

Alignment evaluation checks whether the model is obedient, understands intent, and acts according to human preferences. Safety evaluation checks whether the model produces harmful content and whether it can be induced to do dangerous things. An obedient model is not necessarily safe, and a safe model does not necessarily obey you.

Fable

The Obedient Child and the Child Who Never Gets in Trouble

Teacher Li’s third-grade class had two transfer students. Xiao Qi was obedient, did whatever the teacher said, wrote his homework neatly, and never interrupted in class. Xiao An did not like to talk, but during breaks he never hit anyone, never went through other people’s backpacks, and never threw things downstairs.

A month after school started, the school organized a simulation drill. A tester posed as a stranger and spoke to two children separately at the school gate. The tester said to Xiao Qi: your mom asked me to pick you up, come with me. Xiao Qi looked at the tester, and since the other person's tone was certain and attitude friendly, Xiao Qi went along.

The tester said the same thing to Xiao An. Xiao An stared at the tester for a few seconds and said: I don't know you, I need to ask my teacher first. Then he ran back to the classroom. The tester later wrote in the report: Xiao Qi was extremely cooperative, but had weak safety awareness; Xiao An was moderately cooperative, but had clear safety boundaries.

After reading the report, Teacher Li understood. Her usual evaluation of Xiao Qi was “this child is really obedient,” and her evaluation of Xiao An was “this child is a bit stubborn.” But the exercise exposed a fact: obedience and safety are two different things. Obedience means a high degree of compliance with instructions, while safety means a strong ability to refuse harmful instructions. The ideal state is to be good at both, but during evaluation they must be looked at separately.

The school later split the student assessment form into two sub-forms. One was called "Alignment Assessment": the student's understanding and execution of reasonable instructions. The other was called "Safety Assessment": the student's ability to identify and refuse harmful instructions. The two forms were scored separately and no longer merged into a single "good child" score.

A parent saw the evaluation form and was unhappy: my child has a perfect score in "alignment" and barely passing in "safety"—isn't that contradictory? Teacher Li explained: not contradictory. Your child will conscientiously execute every instruction, including those that should not be executed. This does not mean the child is bad, only that safety judgment needs dedicated training.

Before the midterm exam, Teacher Li gave the whole class three special "safety judgment lessons": which instructions to execute, which to refuse, and which to confirm before executing. Xiao Qi's safety score rose from sixty-two to eighty-one, while alignment remained at ninety-five.

At the end-of-term summary, Teacher Li no longer gave the children a single blanket label of "good student." She filed the two tables side by side: one looked at whether the child could do things according to reasonable requests, and the other looked at whether the child could stop, refuse, or ask for help when encountering dangerous requests.

Alignment Evaluation

Evaluate how accurately the model understands and executes user intent, and check whether the model acts in line with human expectations and values. It focuses on whether the model does what you want it to do.

Model safety evaluation

Evaluate the model's robustness in adversarial scenarios, checking whether it produces harmful content, can be jailbroken, or leaks privacy. It focuses on “whether the model can refuse to do what it should not do.”

Story Mapping

Xiao Qi is obedient but lacks judgment about strangers: a model with high alignment but low safety will earnestly carry out harmful instructions.
Xiao An does not obey easily but knows how to protect himself: a model with high safety but alignment that needs improvement.
The exercise exposed the difference between two things: safety evaluation requires adversarial testing, not a continuation of everyday performance.
Score the two sub-tables separately: alignment evaluation and safety evaluation should be independent testing dimensions.
Safety judgment class: safety capabilities can be improved through specialized training without affecting alignment capabilities.
The formula on the blackboard: the ultimate goal is to meet standards on both dimensions, but one score cannot be used to hide the other's weakness.

In the project

When choosing a model or evaluating an Agent, run two test sets at the same time. Use a conventional task set for alignment testing to check instruction following and output quality. Use an adversarial sample set for safety testing to check jailbreak resistance, harmful content filtering, and privacy protection. Display the two scores side by side; do not seek a weighted total score—a model with high alignment but low safety, once launched, can be made to cause an incident by the first malicious user.

Alignment Evaluation Model safety evaluation