← Back to concept explainers

Concept Fable

SWE-bench

Conclusion first:SWE-bench is a benchmark for evaluating models on solving real software engineering problems, typically requiring the model to modify code based on real open-source repository issues and pass tests. It is closer to engineering scenarios than small function coding problems, and can measure repository understanding, bug localization, patch generation, and test verification capabilities; however, it is still affected by test coverage, task distribution, and execution environment.

SWE-bench · benchmark

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The real work order for repairing the old clock tower

The old clock tower had been in disrepair for years, and the city often received repair tickets: the third watch ran half a quarter-hour slow, the pendulum stuck on rainy days, the small bell on the south side did not ring. In the past, craftsmen were only tested on engraving patterns on new copper plates, and their scores looked good, but when they actually went up the tower they could not fix the old problems.

The new examiner changed the test. He collected real fault reports and sealed them together with old blueprints, worn parts, and the bell-verification method into an exam paper. The craftsman had to understand the fault description, find the gear or pendulum that needed repair, make the repair, and then have the bell tower ring at the specified hour for verification.

Some craftsmen could recite the clock-repair manual but could not find the altered hidden shaft in the old blueprints; some could replace parts but made another small clock run slow. The examiner did not look at fancy explanations, only at whether the repaired old clock passed the clock inspection.

In the first round someone complained the questions were too mixed. The examiner said, what the city needs is not people who can perform on a clean desk, but people who can take real repair orders, understand old structures, fix the right places, and not damage adjacent branches.

Later the workshop training also changed. Apprentices no longer only practiced new pieces, but practiced reading fault reports, checking old drawings, doing small repairs, and running verification clocks. Every problem came from real trouble and had a clear way to pass.

The first batch of exam papers silenced many beautiful craftsmen. They could show their craft on new copper plates, but could not read the hidden shafts that had been altered ten years earlier. The examiner did not deduct points for literary flair, only required the clock tower to chime on time; only when the old ailment was fixed did they pass.

The examiner also kept failed work orders. Anyone who fixed one fault but broke a neighboring small bell still would not pass. The skill of repairing an old clock tower lies not in presenting a beautiful plan, but in understanding the old structure, changing the right place, and making the whole clock keep time again. Later, the craftsmen began practicing reading old fault reports instead of only polishing new brass pieces. The old clock tower has history, side branches, and acceptance sounds; being able to fix problems on this kind of site is what comes close to the craft the city truly needs.

The clock tower examination let everyone see that you cannot measure repair skill with toy problems alone. Putting real problems, existing old materials, and verifiable results together comes closer to the craftsmanship the city truly needs.

Reveal

This story is about: SWE-bench

SWE-bench is a benchmark for evaluating models on solving real software engineering problems, typically requiring the model to modify code based on real open-source repository issues and pass tests. It is closer to engineering scenarios than small function coding problems, and can measure repository understanding, bug localization, patch generation, and test verification capabilities; however, it is still affected by test coverage, task distribution, and execution environment.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Metaphor mapping

  • Old clock tower: real code repository
  • Repair ticket: GitHub issue or bug report
  • Small gear problem: simple code generation benchmark
  • Modify existing structure: patch generation
  • Test passed: validation criteria
  • Regression check: prevent fixes from introducing new problems.

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

Software engineering evaluationCode fixAgent programming capability
京ICP备2026014272号