Concept Contrast Fable

Prompt tuning vs. fine-tuning: one teaches the model a secret code, the other gives the model a new brain

Conclusion first:Both terms include "fine-tuning," but what they change is on a completely different level. Prompt tuning trains a set of soft prompt vectors at the input layer, leaving the model's own parameters untouched; fine-tuning updates some or all of the model's internal weight parameters. Prompt tuning changes "what signal is handed to the model," while fine-tuning changes "how the model processes the signal once it receives it." The cost differs by dozens of times, and the applicable scenarios are entirely different.

Both terms include "fine-tuning," but what they change is on a completely different level. Prompt tuning trains a set of soft prompt vectors at the input layer, leaving the model's own parameters untouched; fine-tuning updates some or all of the model's internal weight parameters. Prompt tuning changes "what signal is handed to the model," while fine-tuning changes "how the model processes the signal once it receives it." The cost differs by dozens of times, and the applicable scenarios are entirely different.

Fable

The conductor and the composer: two different ways of changing

A symphony orchestra took on two kinds of tasks at the same time. The first was preparing a performance plan for a new piece: the conductor receives a new piece specified by the organizer and needs to find the most suitable conducting gestures during rehearsal—which section starts on which bar, how large the crescendo should be, how long the closing signal should last. The orchestra members stay the same; what changes is how the conductor conveys instructions.

The second type of task is adaptation: the organizers felt the brass parts of the original piece were too sharp and asked for the entire arrangement to be rewritten. The composer did not keep adjusting conducting gestures, but directly rewrote the score: assigning the brass melody lines to the strings, adjusting the harmonic structure, and even deleting some passages and recomposing them. This time what changed was the piece itself.

Both types of tasks were written as "performance adjustments" on management's scheduling sheet, but the orchestra's investment was completely different. For the conductor to change gestures required only three rehearsals, with the orchestra showing up normally, and a budget of tens of thousands. For the composer to rearrange the orchestration required re-dividing parts, re-recording demos, and re-proofreading dozens of pages of scores, with a cycle measured in weeks and a budget more than ten times higher.

Once the organizers confused two types of tasks. The effect they wanted was for the music to sound "softer," but the request they made was to "change the performance approach." The conductor spent two rehearsals adjusting gestures—making the strings more prominent and suppressing the brass entrance timing. It did sound somewhat softer, but the brass's sharp timbre itself had not changed; with a different conductor it might revert.

The orchestra manager later set a rule: if only the performance effect of the same piece needs to change for a specific occasion, ask the conductor to adjust gestures. If the piece needs to sound different on any occasion and under any conductor, then ask the composer to change the score. The cost difference between the two directions is huge, and they cannot be written under the same work order name on the schedule.

The orchestra later wrote this set of judgments into a flowchart on the rehearsal room wall: if the original piece itself is suitable and only this performance needs a different breath point, dynamic, or entrance rhythm, let the conductor adjust the gestures. If the original composition is already unsuitable for the target occasion, and even ten different conductors could only slightly conceal it, ask the composer to rewrite the score.

At the very bottom of the flowchart there is another line in red: first listen to whether the problem is at the console or in the score. The former can be changed quickly, at low cost, and switched between performances; the latter is changed deeply, takes longer, and leaves a new version. Mixing the two kinds of work orders will at best cost extra rehearsal fees, and at worst send the whole performance onstage with the wrong score.

Prompt fine-tuning

Train a set of learnable soft prompt vectors at the model input layer to guide the model toward better performance on a specific task, while keeping all parameters of the model itself unchanged.

Fine-tuning

On the basis of a pretrained model, update some or all of the model's weight parameters with task-specific data so that the model acquires new knowledge or adjusts its behavior patterns.

Story Mapping

The conductor adjusts gestures to make the same piece softer: prompt tuning, changing how input is guided without touching model parameters
The composer rewrites the score and changes the piece itself: fine-tuning, updating model weight parameters
The organizers conflated the two types of tasks: writing the same work order name on the schedule led to misallocated resources
Want a softer sound, find the conductor; want to change the tune, find the composer: first determine whether what is missing is guidance or knowledge
A cost difference of over ten times, and completely different use cases: prompt tuning cannot replace fine-tuning, and fine-tuning shouldn't be wasted on scenarios that only need guidance

Falling into the model adaptation plan

When receiving a customer adaptation request, first fill out a decision table: does the model already have enough knowledge in this domain? If so, and it just needs a better way of asking, use prompt tuning. If the model lacks relevant knowledge in this domain or the style is completely off, go with fine-tuning. Prompt tuning has low training cost and flexible deployment but cannot add new knowledge; fine-tuning can change model capabilities but requires significant time and GPU investment. Clearly distinguish these two rows in the procurement list to avoid spending fine-tuning money on prompt-tuning work.

Prompt fine-tuning Fine-tuning