Multimodal RAG

In the past, the old bridge team repaired bridges by looking only at the written archives left by the Ministry of Works. The archives recorded pier dimensions, the age of wooden beams, and the time of the last repair. When the bridge was small and the water shallow, reading the archives was enough.

Later, the city needed to build a bridge across the river. Text archives still existed, but the site also had crack photos, survey maps, old craftsmen's annotations, water sounds recorded by boatmen, and rainy-season videos.

The captain first had the clerk transcribe everything into text. Fine cracks in photos were written as 'has cracks,' slopes on drawings as 'slightly slanted,' and the anomalous rhythm in the sound of water couldn't be transcribed by anyone at all.

The engineers built the plan according to these words. The moment the first beam was lifted, they found the load direction did not match the drawings. The words had not lied—they had simply lost too much shape, position and timing.

Someone suggested trusting only photographs. But photographs can't show history, blueprints can't show tides, and recordings can't reveal material aging. Each kind of material holds only part of the problem.

A surveyor indexed archives, drawings, photos, videos, and recordings separately, then let the same question seek evidence across different materials, and finally aligned the evidence to the bridge's parts and timeline.

When the team lead asked "why is the east pier sinking," the system simultaneously produced old maintenance records, recent photos of cracks, rainy-season video, and water flow measurements. The answer finally took shape.

The bridge repair crew later understood that a bridge does not live only in text. To make the materials useful, evidence from different modalities had to be findable, aligned and citable together. Then a trickier job came along. On the surface it looked like an old problem magnified, but it actually forced out every gap hidden in the process: who takes over first, who can make changes, how to return to the previous step after an error—places that used to get by on familiarity and tacit understanding now all had to be made explicit. The person in charge did not urge everyone to be more careful again. Instead, he walked the site once more. He had people record every handover, every failure and every recovery, noting which places were merely slow, which could harm the results, and which would be hard to trace back once wrong. After the new rules were rolled out, the first few days were not smooth. Some complained about an extra check; some felt their old experience was being insulted. But when the next anomaly appeared, for the first time everyone could follow the traces to find where the problem had occurred, instead of gathering around the result and guessing at one another.