Concept Fable

Catastrophic forgetting

Conclusion first:Catastrophic Forgetting is the phenomenon in which a neural network, after fine-tuning on new data or a new task, significantly loses previously learned capabilities. When enterprises fine-tune LLMs, overly narrow data, too high a learning rate, or too many training epochs can all cause degradation of general capabilities, safety alignment, or existing business capabilities. Mitigation methods include mixing in old data, regression evaluation, a smaller learning rate, freezing some layers, regularization, and continuous evaluation.

Catastrophic Forgetting · Training

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Once the new rules were memorized, the old paths were all forgotten

The post station messenger Ating had memorized the northern mountain passes, and was later assigned to specialize in the southern waterways. The old stationmaster initially relied on the newly practiced route; with familiar people and familiar roads, few mistakes were made when the days were small. The stationmaster discovered that Ating learning the southern route did not mean capability had increased; if the new training overwrote the northern route memory, the model had merely traded one skill for another.

Later, Ating ran very fast on the South Road, but when he returned to the North Road he forgot which bridge would freeze over in winter. The more concentrated the samples for the new task, the faster performance on the old route declined; looking only at South Road results would hide the catastrophic forgetting that had already occurred behind the average score.

The first fix was urgent: the stationmaster had him practice the south road ten more times, thinking that mastering one route was better than wavering between two. Repeatedly practicing the south road to mastery only improved the single-item score, while the key fork on the north road was diluted by the new memory. But new problems followed, everyone only stared at the most easily seen places, and small deviations in the distance accumulated deeper and deeper.

He trained on a mix of old and new routes, retained key old samples, periodically replayed them, and evaluated the two task types separately; each learning session checked whether new capabilities came at the cost of degrading old ones.

The old stationmaster mixed old and new routes in the schedule and kept several key fork diagrams, requiring a run back along old routes to verify each time a new route was learned. This step was troublesome and at first slowed the work; but after several rounds, rework decreased, and masters no longer changed rules on a whim. The station guarded old and new routes together, and the gains from learning no longer came at the cost of erasing existing capabilities.

From then on, adding new capabilities no longer came at the cost of erasing old memories; continual learning must simultaneously protect the gains on new tasks and the baseline on old tasks.

Only then did A-Ting understand: if learning a new skill wipes out the old one, the relay station is merely swapping one risk for another. From then on, when gaining new knowledge, it must retain the memories from the past that are truly useful.

Concept Anchor

This story is about: catastrophic forgetting

Catastrophic Forgetting is the phenomenon in which a neural network, after fine-tuning on new data or a new task, significantly loses previously learned capabilities. When enterprises fine-tune LLMs, overly narrow data, too high a learning rate, or too many training epochs can all cause degradation of general capabilities, safety alignment, or existing business capabilities. Mitigation methods include mixing in old data, regression evaluation, a smaller learning rate, freezing some layers, regularization, and continuous evaluation.

What matters is that it is not just a term, but a structural problem that repeatedly appears in real AI / Agent systems. Understanding it lets you judge when to add a model, when to change a process, and when to add governance.

Story Mapping

New city district route: new fine-tuning tasks or domain data
Forgetting the old path: degradation of existing capabilities
Cyclical review of old paths: replay/mix original or representative data
Key route exam question: regression evaluation set
New district improves but old city fails: capability retention must be checked before launch

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

continual learningSFTpre-trainingregularization