← Back to concept explainers

Concept Fable

Data mixing

Conclusion first:Data mixing is the strategy of combining different data sources in proportions during pretraining or fine-tuning. When the proportions of code, math, Chinese, safety, customer service, industry documents, and other data differ, the capabilities the model ultimately excels at and the biases it develops will also differ.

Data Mixing · Training

Read the story first. There is no rush to give a definition here; let the question grow on its own first.

Fable

The proportions of herbs in a pharmacy's formula

The South City apothecary had a large medicine pot, specially for training apprentices to identify prescriptions. The master put samples of mountain herbs, seaside shells, common pills and powders from the city, and several rare poisonous herbs into the library, and had the apprentices identify them daily according to prescriptions.

At first, the master, looking to save effort, dumped in baskets full of the most common cold remedies and trauma remedies. The apprentice learned fast, and whenever an ordinary customer came in, he could fill the prescription immediately. But when a rare symptom appeared, he either failed to recognize it or mistook a rare poisonous herb for an ordinary heat-clearing one, nearly causing disaster. The first time, the master looked only at the overall accuracy rate; ordinary prescriptions were the majority, so the numbers looked great. Only when the rare poisonous herb caused an error did he realize that the minority but high-risk samples had been hidden by the average.

Someone suggested dumping all the rare herbs in. A few months later a new problem appeared: the apprentices memorized strange prescriptions every day, yet couldn't even be sure about the most common cough and cold. The shop seemed more knowledgeable, but its actual counter service became less reliable.

Only then did the master rearrange the medicine storage. He placed common formulas according to the real proportions of visitors, and reserved fixed practice slots for high-risk, high-cost medicinal materials; overly similar medicines had to be compared on alternate days, and rare but critical medicines had to be given heavier assessments. After each practice session, he also looked at which types of formulas the apprentices often got wrong, then adjusted the next batch of samples. Rearranging the medicine storage was not a one-time task either. After winter set in, cough and asthma visitors increased, and in spring eczema became more common; if the proportions in the medicine pot did not move with the seasons, the apprentices' feel would drift again.

The apprentice discovered that the materials in the medicine pot were not better the more there were, nor better the rarer they were. The source, proportion, repetition count, and difficulty of the materials together shaped his feel. If one ingredient was added too much, the other flavors were masked; if a certain type of illness never appeared, he would be blind when standing at the counter. Some sources also conflicted with each other. In the city, pills and powders were written in neat forms, while mountain herbs had chaotic names; without cleaning and labeling, the apprentice would mistake differences for medicinal properties.

The apothecary finally left a rule: compounding a practice remedy is not grabbing a handful from the warehouse at random, but balancing the scales among various sources. When the scales are balanced correctly, the apprentice neither over-specializes nor gets led astray by a few noisy samples.

Reveal

This story is about: Data mixing

Data mixing is the strategy of combining different data sources in proportions during pretraining or fine-tuning. When the proportions of code, math, Chinese, safety, customer service, industry documents, and other data differ, the capabilities the model ultimately excels at and the biases it develops will also differ.

So it is not simply "the more data, the better." When enterprises continue training, they need to bring data sources, sampling weights, and the risks of excess and insufficiency into evaluation, and use the results to adjust the proportions in turn.

Metaphor mapping

  • Different ingredients = different data sources
  • Recipe proportions = sampling weights and data mixing ratios
  • One flavor too strong = a certain capability or bias being amplified
  • Tasting = evaluation feedback
  • Readjusting the recipe = adjusting data ratios based on evaluation

Andy Ren Xu's judgment

The practical value of this concept is that it helps you break "AI features that look smart" into deliverable, verifiable, and governable units of work.

pre-trainingdata curationdomain weights
京ICP备2026014272号