Tokenization

The old post office bought a sorting machine that could not directly read a person's whole letter. It only recognized a fixed set of wooden pieces, each carved with a number. Before a letter entered the machine, it had to be cut into these wooden pieces; the numbers the machine spat out also had to be reassembled into sentences people could read.

At first the letters were simple, and the slicing master could just cut by word. Later, letters contained foreign languages, personal names, item numbers, table symbols, smiley faces, and long addresses. Cutting by word would encounter unseen words, while cutting by character was too fragmented, and the worktable quickly piled up with wood shavings.

Whenever the master encountered an unfamiliar word, he threw the whole block into the exception box. The exception box grew fuller and fuller, and the machine could not process the new product names, order numbers, and proper terms that customers cared about most. The courier complained: the more important something new is, the less it can get through.

He then changed to cutting every character apart. The machine could always read it, but it required too many wood chips, processing slowed down, and before a long letter was finished, the chip box was full. The accountant discovered that the more finely the same letter was cut, the higher the cost and the less content could fit.

The post office discovered that the way of splitting itself affects what the machine can see, how much it costs, how much long letters can hold, and whether certain languages are disadvantaged. So they trained a set of splitting rules: common fragments are kept together, rare words are split into combinable small pieces, and spaces, punctuation, and special symbols also have clear handling methods.

The first version of the rules leaned toward local dialect, and out-of-town names got chopped to pieces. The master then put letters from various regions into the practice pile, letting common combinations grow into larger wooden pieces on their own, while rare combinations could still be split and reassembled. Slicing was no longer a random cut, but a reusable convention.

Later, the post office replaced its sorting machine, and the old slicing rules could not be mixed in directly. The same text might be cut into different numbers of slices in front of different machines, so both the bookkeepers and the technicians had to recalibrate. The text had not changed, but the way it was cut at the entrance had, and the counting and understanding that followed would change with it.

Later the accounting office tallied the cost of different segmentation methods and found that some languages were always cut into smaller pieces, requiring more wood slips to process the same letter. The post office then included segmentation rules in acceptance: not only whether the machine could read, but also how it segmented, whether segmentation was fair and whether cost was controllable.