Concept Contrast Fable

The big one that turns everything into vectors, and the old master within it who turns only words into vectors

Conclusion first:Embeddings are a general technique—mapping text, images, audio, code, and even entire documents into vector space while preserving semantic similarity. Word embeddings are the earliest form of embeddings, born specifically to model word-level semantics, using static vectors to represent individual words. Treating word embeddings as the whole of embeddings leads one to assume all embeddings are static single-word semantics; treating sentence embeddings as a simple upgrade of word embeddings ignores that modeling at different granularities requires completely different training approaches.

Embeddings are a general technique—mapping text, images, audio, code, and even entire documents into vector space while preserving semantic similarity. Word embeddings are the earliest form of embeddings, born specifically to model word-level semantics, using static vectors to represent individual words. Treating word embeddings as the whole of embeddings leads one to assume all embeddings are static single-word semantics; treating sentence embeddings as a simple upgrade of word embeddings ignores that modeling at different granularities requires completely different training approaches.

Fable

The painting academy upgraded from portraying each character to portraying the entire passage

There was once a group of portraitists in the town. The earliest batch were called word-portraitists, and they did one thing every day: assign a set of coordinates to each character. For example, the character "river," whether it appeared in "Yellow River" or "slippery river," used the same set of numbers across all books in the library. The coordinates came from statistics: the frequency with which this character appeared adjacent to other characters in all the town's books. "River" and "water" often sat next to each other, so their coordinates were close; "river" and "table" almost never shared a frame, so their coordinates were far apart.

Later, a sentence painter came to town. He did not paint a portrait of a single character; instead, he looked at a whole sentence and assigned a set of coordinates to the entire sentence. "The Yellow River bends north of the city"—the sentence painter's method flowed in one go, swallowing the whole sentence and spitting out a set of numbers. This sentence and "The great river curves past the north of the city" were very close in coordinates, even though the wording differed greatly. "Yellow River" and "Huai River" were close in the character painter's work because both contain the character "river," but in the sentence painter's work they were far apart, because the whole sentences were talking about completely different things.

A teacher built a retrieval system for a library and first used the painter's approach. Searching for "summer rain," the system put the three portraits "summer," "of," and "rain" together and went into the vault to find the book with the nearest coordinates. What came back was "Winter Snowflakes"—"winter" and "summer" were close in coordinates, "snowflakes" and "rain" were also close, and "of" was identical. The teacher shook his head: the words were not wrong, but the book was completely wrong.

Switching to the sentence painter's method, searching for "summer rain" used the portrait of the whole sentence. What came close was no longer "Winter Snowflakes" but "Jiangnan Plum Rain Season" and "June Thunderstorm Preparedness Guide." Pleased as the teacher was, he began to ask: how long should a sentence be to be just right? Too short, and it degrades into the character painter's route; too long, and the painted coordinates blur into a single smear. The sentence painter's answer was a self-made ruler—the length of the sentence should just enclose one complete meaning.

More portrait artists poured into the town. Some painted portraits from photos, some from voice waveforms, some from a piece of code. The calligrapher-painter realized he was just a small studio within a larger academy of painting. All portrait artists used the same mathematical tool—high-dimensional vectors—but their training corpora and optimization objectives were completely different.

The library's retrieval system later hung three kinds of portraitists: word-portraitists for exact matching, so rare terms were not missed; sentence-portraitists for semantic search, so concepts could be found in paragraphs with different wording; and picture-portraitists for cross-modality, so photographing an illustration found the corresponding book page.

The painting academy's apprentice rules gained new entries: the character painter is an inherited craft, inferring semantics from word co-occurrence frequencies. The sentence painter was the previous generation's invention, inferring semantics from the context of context. The picture painter, sound painter, and code painter are the grandchildren, bringing portraits from text to everything that can become a sequence. The same mathematical toolkit unfolds at different granularities, each capturing similarities at different levels.

Embed

A general technique for mapping arbitrary objects into a continuous vector space. The input can be words, sentences, paragraphs, images, audio, or even graph structures, and the output is a vector representation that preserves semantic similarity. It is one of the most fundamental abstractions in modern deep learning.

Word embeddings

Early word-level applications of embedding technology, such as Word2Vec and GloVe, generated static vectors for each word. The same word used the same vector in all contexts; the advantage was simplicity and efficiency, and the disadvantage was the inability to distinguish different meanings of polysemous words.

Story Mapping

Word artist: word embeddings, generating a static vector for each word that stays the same across different contexts
Sentence Painter: sentence and paragraph embeddings, generating context-relevant vectors for an entire passage
'Summer rain' matched to 'winter snowflakes': static word embeddings look only at literal co-occurrence and lose semantics
The sentence painter found "Jiangnan Plum Rain Season": context-sensitive embeddings capture true intent better
Picture artists, audio-visual artists, code artists: embeddings expand from text to multimodal
Three parallel systems in the archive: embeddings at different granularities each do their job, and only combined use can cover all scenarios

This comes down to vector retrieval and semantic search design

Word embeddings are still useful for keyword-level exact matching—highly efficient when searching terms, abbreviations, or specific entities. For semantic search and intent matching, use sentence- or document-level embedding models, which can capture true meaning across literal differences. Multimodal scenarios require embedding models of the corresponding type; text vector spaces and image vector spaces have their own coordinate systems and cannot be mixed. For cost-sensitive scenarios, first assess whether static word embeddings can meet the need before considering a pretrained sentence embedding model—the inference budget saved may be enough to cover an extra feature.

Embed Word embeddings