Fable
Another tokenization method did not steal the old ledger's name
The old print shop wanted to carve movable type for the whole city. At first they carved one piece for every character. Common characters were easy, but rare characters and foreign names kept piling up, and the type cabinet was nearly full. Finding a character took longer than printing a page.
The master switched to the smallest character blocks, carving each horizontal, vertical, left-falling, and right-falling stroke separately. This way any character could be assembled, but typesetting was too slow—even a common word required a dozen blocks. The apprentice ran his legs off every day, and barely a few pages were printed.
The workshop owner observed for half a month and found that some character blocks always appeared in pairs: a certain radical often sat next to a certain phonetic component, certain two syllables often formed a place name, and certain characters in a shop name were always linked together. So he combined the most common pair into a new character block and put it in the character cabinet.
Each time they merged, typesetting got a little faster. But the print shop owner did not carve every word into a whole block; he picked only the most frequent combinations. Rare ones were still split and assembled, while common ones used larger blocks. The type cabinet gradually formed a mixed set of materials: small strokes, common radicals, and high-frequency word blocks.
The first version merged too greedily; certain popular firm names were carved into large blocks that took up half a cabinet once the season passed. The workshop owner changed to looking at frequency by long-term ledger, first merging the most stable adjacent combinations, then gradually adding new blocks. A character cabinet cannot chase only momentary excitement.
When the print shop moved, a new apprentice used a different set of type cases to set an old book, and the page count suddenly increased. It turned out that in the old case, a business name was a single block, but the new case split it into two. The characters hadn't changed, but the blocking method had, so the count and typesetting rhythm changed.
The master reminded them that the secret of movable type was not just which characters existed, but how to cut continuous text into blocks. Cut too finely, and efficiency drops; cut too coarsely, and the type cabinet swells. A good segmentation grows out of common collocations, balancing coverage and typesetting efficiency.
Later, when the print shop received foreign-language commercial orders, the owner did not rush to carve a whole block for every foreign name. He first looked at which combinations appeared often and which were just one-off signposts. The type cabinet could therefore take on new work without being stuffed full of rare large blocks. The more stable the cutting method, the more room there was for typesetting.