Concept Contrast Fable

Those found by relationship graph and those found by multiple data forms

Conclusion first:Graph RAG adds entity and relationship structures on top of text to improve retrieval, while multimodal RAG adds support for multiple data formats such as images and audio. The two have different sources of complexity and solve different types of information silos.

Graph RAG adds entity and relationship structures on top of text to improve retrieval, while multimodal RAG adds support for multiple data formats such as images and audio. The two have different sources of complexity and solve different types of information silos.

Fable

Two book bureaus: one draws relationship diagrams, one collects diagrams and keeps accounts

The county had two bookshops, one in the east and one in the west, both claiming they could look up all kinds of information. The east shop drew a large map of personal relationships and business dealings and hung it on the wall; when looking something up, they first found the map, then followed it to the source. The west shop put texts, illustrations and ledgers all in the same cabinet, and searched them together whenever a query came in.

The East Bureau first showed its stuff when someone asked: all the firms that did business with Manager Wang last year—what happened to them afterward? The East Bureau's manager traced the relationship graph all the way down and pulled out the follow-up transaction records of twelve firms. The asker was very satisfied and said the graph was truly useful.

But there were things the East Bureau could not find either. Once someone brought a fabric pattern and asked whether this design had any historical record. The East Bureau searched the entire relationship graph, but no node connected to the pattern, and all records were text. This question became an unsolved case at the East Bureau.

The West Bureau first showed its ability when someone brought an illustrated product catalog to look up performance parameters. The West Bureau ingested the illustrations and text together, and searching by image directly found the corresponding parameter table. The shopkeeper said this was true multimodality.

But the West Bureau ran into its own problem. Once, when investigating the history of a certain firm, it needed to connect the timeline and transaction amounts in the written records. What the West Bureau retrieved was a pile of fragments, and the relationships between the fragments were not structured, so they could only be pieced together manually. The shopkeeper said, you search comprehensively but you do not connect them; they cannot be linked.

The two bureaus later did a joint demonstration, and a complex query came in: among merchants connected to a certain cloth merchant, had anyone used a particular pattern, and in which dynasty did this pattern first appear in historical records. The East Bureau first found the merchant relationships, the West Bureau then searched the pattern and dynasty, and only by combining both sides could they answer.

After the demonstration, someone asked the two shopkeepers: which of you is better? Both said their own direction was right, and the problem was that they could not solve what the other solved. An old reader beside them said one thing: what you are arguing about is not who is better, but whether your readers need relationships or multimodality—if they need relationships, you keep drawing graphs; if they need multimodality, you keep collecting images and drawings; but for queries that need both relationships and multimodality, neither of you alone can handle them.

Graph RAG

Adding entity and relationship graph structures on top of text retrieval-augmented generation enables answering cross-document relational questions. Building the graph is costly, but it can improve results for queries involving multi-hop reasoning.

Multimodal RAG

Retrieval-augmented generation that supports text, images, audio, and other data formats. It can handle unstructured multimodal content, but the relational structure within the text still depends on the quality of text retrieval itself.

Story Mapping

East Bureau maps people relationships and transaction graphs: Graph RAG, using graph structures to improve cross-document association queries
The western bureau put text, images, and ledgers all together for search: multimodal RAG, unified handling of multiple data formats
Pattern history left unsolved at the East Bureau: Graph RAG cannot handle non-text modalities
The Western Bureau search timeline and transaction amounts are all fragments: multimodal RAG lacks relational structure, and fragments cannot automatically connect
It takes two rounds combined to answer a complex query: two RAG approaches solve problems in different dimensions, and only together can they cover complex scenarios
Whether the reader needs relationships or multimodality: first determine the main source of query complexity, then choose the corresponding approach

Applying this to RAG architecture investment decisions

When deciding whether to build Graph RAG or multimodal RAG first, look first at where your queries are mainly difficult. If queries require finding relationships across multiple documents and answers involve multi-hop reasoning, build Graph RAG first. If queries require handling multimodal content such as images, audio and scanned documents, build multimodal RAG first. If you need both, then before building either, first assess whether your text retrieval foundation is good enough—both multimodality and graphs depend on the quality of the underlying text chunking and indexing. If the foundation is not solid, no matter how complex the upper layer is, it is a castle in the air.

Graph-enhanced retrieval generation Multimodal RAG