Fable
The warehouse switched to a measuring bucket that could hold half a unit of goods
The silversmith shop needed to pack a large batch of silver grains into small boxes, and originally each grain was placed separately in a large compartment. At first, the silversmith relied on a more compartment-efficient counting method, familiar and routine, and when the business was small, mistakes were rare. The silversmith discovered that the real challenge was not writing the numbers smaller, but preserving the numerical information the model still needs to function within limited VRAM.
Later there were not enough boxes, and moving was slow. The smaller the boxes, the sharper the trade-off between precision and capacity; crude merging saves space but may let errors in key weights accumulate into noticeable quality loss.
The first fix was rushed: the apprentice wanted to merge silver grains casually, but the weighing error became too large. Compressing all silver grains into the same short mark did save space, but it made the error in key values uncontrollable. Yet new problems followed: everyone stared only at the most visible places, and small deviations in the distance accumulated deeper and deeper.
The old master compressed weights by group range, retained the necessary scaling information, and used calibration weights to compare errors before and after quantization; bitsandbytes makes this low-bit loading and training process practical.
The old master replaced common weights with short marks, kept enough gradations for key large numbers, and left calibration weights for conversions. This step was troublesome and initially slowed the work; but after a few rounds, rework decreased, and the master no longer changed rules on a whim. The warehouse gained VRAM space, but did not treat key precision as margin that could be compressed indefinitely.
The VRAM saved from then on can be traded for a larger model or more concurrency, but every quantization must be proven worthwhile through measured task quality and throughput.
The silver grains fit, and the accounts largely matched. From then on, saving space could rely on tighter notation, but one had to know where not to over-compress.