Concept Fable

Layer-wise learning rates

Conclusion first:Layer-wise Learning Rate means setting different learning rates for different model layers during fine-tuning, commonly smaller for lower layers and larger for higher or task-related layers. It helps preserve general representations while letting the model adapt to new tasks, reducing the risk of catastrophic forgetting and overfitting. It deserves particular attention in enterprise fine-tuning with small data, specialized-domain tasks, or transfer learning.

Layer-wise Learning Rate · Fine-tuning

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

Light touch-up on the old wall, heavy repaint on the new door

While repairing an ancient pagoda, the master craftsman found that the bluestone at the base had endured years of wind and rain, while the new tiles at the top had just been laid. The young craftsmen argued for renovating the whole pagoda together, using the same hammer and the same force, thinking this was fastest and fairest.

A few days later, the loose new tiles at the top of the tower still had not adhered, while the old stones at the base had cracked from the shaking. The foreman understood that different places could bear changes differently; applying the same repair to every position would not produce the same result.

He changed to testing layer by layer on a small scale: for the tower base, only patching cracks and reinforcing corners; for the middle layer, replacing damaged components; for the tower top, recalibrating the tile rows. Every layer's changes were recorded in the ledger, and if cracks were found to widen, he reverted to the previous step.

By early autumn, the ancient pagoda had both preserved its foundation and replaced its roof. The master craftsman told the apprentice that to repair a pagoda with history, one must first distinguish where to touch lightly and where bold adjustments are possible.

Concept Anchor

This story is about: layer-wise learning rates

Layer-wise Learning Rate means setting different learning rates for different model layers during fine-tuning, commonly smaller for lower layers and larger for higher or task-related layers. It helps preserve general representations while letting the model adapt to new tasks, reducing the risk of catastrophic forgetting and overfitting. It deserves particular attention in enterprise fine-tuning with small data, specialized-domain tasks, or transfer learning.

Its value lies in grounding an abstract term in observable mechanisms, constraints, and trade-offs, making it easier to judge the scope of applicability in real systems.

Story Mapping

Old apothecary's foundation: the model's underlying general representation
Front hall: task-related high-level layers or output heads
Polishing with the same force: all layers use the same learning rate
Layered Processing: layer-wise learning rate
Retain old knowledge while adapting to new processes: stable transfer and fine-tuning

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

discriminative fine-tuningcatastrophic forgettingULMFiT