← Back to Concept Contrast

Concept Contrast Fable

Pre-training and fine-tuning: one lays the foundation, the other renovates the room

Conclusion first:Pretraining and fine-tuning are two adjacent stages in the model training pipeline, often lumped together as "training." Pretraining builds foundational capabilities on massive general data, while fine-tuning adjusts behavior on task-specific data. Confusing the two leads to choosing the wrong base model, misestimating budgets, and misidentifying which stage a capability bottleneck actually sits in.

Pretraining and fine-tuning are two adjacent stages in the model training pipeline, often lumped together as "training." Pretraining builds foundational capabilities on massive general data, while fine-tuning adjusts behavior on task-specific data. Confusing the two leads to choosing the wrong base model, misestimating budgets, and misidentifying which stage a capability bottleneck actually sits in.

Fable

The conservatory's foundation training and performance rehearsals

A music conservatory's training system is divided into two major phases. In the first three years, all students take the same courses: ear training, music theory, music history, and basic piano and vocal skills. Whether they later want to be a soloist, an accompanist or a conductor, during this period they are all building the same foundation. Looking back after graduation, no one knew during those three years what they would ultimately specialize in, but anyone with a weak foundation could not go far in any direction afterward.

In the fourth year, students branch into directions. They choose a major based on their aspirations and circumstances, entering a specialized rehearsal season. Solo-performance students grind through concertos daily, chamber music students extensively refine ensemble coordination, and composition students begin writing their own works and seeking orchestras for tryouts. The training methods and content of the rehearsal season are completely different from the foundation stage: not casting a wide net, but repeatedly polishing toward one clear goal.

The academy once tried a reform: let first-year students take foundation courses and rehearsal courses at the same time. The academic affairs office thought, "Earlier exposure to real practice means faster learning." The result was that students who couldn't even play their etudes were forced into concertos, and people who didn't understand chords were pushed to write compositions. After one semester, foundation courses had widespread failures, and the rehearsal course results were unlistenable.

The academy urgently called a halt and set a rigid rule for course order: those who had not passed the foundation exam in the first three years were not allowed into rehearsal classes. This was not meant to be difficult—it was because the prerequisite assumption of rehearsal class is that the foundation is already in place. A conductor will not stop rehearsal to teach you how to count beats.

Later, a visiting professor came to the academy to share. During the lecture, he drew a thick line on the blackboard, writing "general" on the left and "specialized" on the right. Investment on the left was calculated annually, with high cost and general capabilities as output; investment on the right was calculated weekly, with controllable cost and specialized performance as output. The two budgets were made separately, and the two sets of evaluation standards were set separately.

The academy wrote this line into its teaching evaluation system. Assessment at the foundation stage looks only at coverage and solidity of basic skills, not specialized performance. Assessment at the rehearsal stage looks only at the completeness and artistic quality of the target repertoire, without going back to check basic skills. Each stage has its own standards, eliminating the confusion of using rehearsal performance to grade the foundation or using foundation standards to block rehearsal results.

Two years after the reform, employment data showed an interesting change: students who had been streamed from the foundation stage had shorter adaptation periods when switching direction. The reason was that the foundation had not been compressed; they entered the new direction with complete general capabilities and only needed to rehearse. Those whose foundation had been skipped or rushed had to start from scratch when switching direction.

Pretraining

Training a model on massive general data to build basic language ability and world knowledge is the full-set, high-cost, one-time foundation-building stage.

Fine-tuning

Adjusting model behavior on top of a pretrained model using task-specific data; cost is relatively controllable, and it can be performed multiple times for different scenarios.

Story Mapping

  • The same foundation course for the first three years: pretraining, building basic capabilities on massive general data
  • Year four direction rehearsal season: fine-tuning, adjusting behavior on task-specific data
  • Doing foundation work and rehearsals simultaneously in the first year led to widespread failure: skipping pretraining and going straight to task fine-tuning is not viable
  • You cannot enter rehearsal class without passing the foundation exam: pretraining quality is a prerequisite for fine-tuning results
  • Those with a complete foundation adapt faster when changing direction: solid pretraining lets the model transfer more efficiently across different fine-tuning directions

Applying it to an Agent project

When choosing a model, first distinguish two types of needs: those requiring additional general knowledge and foundational capabilities should consider pretraining or selecting the latest pretrained base; those only requiring adjustment of behavior, style, or output format should use fine-tuning. For budget planning, estimate pretraining investment by base model tier, and fine-tuning by task data volume and iteration count. Do not use a fine-tuning budget while expecting pretraining results.

京ICP备2026014272号