Graph RAG: Retrieval Augmented Generation Meets Knowledge Graphs

Standard RAG retrieves chunks of text and hands them to an LLM. Graph RAG retrieves a structured slice of a knowledge graph instead — entities, relationships, and the paths between them. That shift changes what kinds of questions the system can actually answer.

Why vector search runs out of road

Chunk retrieval — A vector database finds text chunks that are semantically similar to a query and stuffs them into the prompt. It works well for lookup questions: "What was the revenue in Q3?" It breaks down for questions that need connecting several facts scattered across different documents.

The multi-hop problem — Ask "Which suppliers does our biggest customer's biggest competitor use?" and a vector search has no reliable way to chain that logic. Each chunk is retrieved independently based on similarity to the query, not based on the relationships between the entities involved.

Lost global context — A single chunk rarely captures how an entity relates to the rest of the document set. A knowledge graph does, because the relationships are stored explicitly rather than implied by proximity in text.

What Graph RAG actually does

Graph RAG builds a knowledge graph from source documents, then retrieves subgraphs relevant to a query instead of flat text chunks.

Entity extraction — An LLM (or a dedicated NER pipeline) reads the source documents and pulls out entities: people, organizations, products, events, dates.

Relationship extraction — The same pass identifies relationships between entities — "acquired," "reports to," "supplies," "located in" — and stores them as edges connecting the entity nodes.

Graph construction — Entities and relationships get written into a graph database (Neo4j, TigerGraph, or a similar store). Each node can carry metadata and a link back to the source text it came from.

Community detection — Many Graph RAG implementations, including Microsoft's original approach, cluster the graph into communities of closely related entities and generate a summary for each cluster. This gives the system a mid-level view between raw entities and the full graph.

Query-time retrieval — At query time, the system identifies relevant entities in the question, walks the graph to pull connected nodes and relationships, and passes that structured context to the LLM alongside or instead of raw text chunks.

Local search vs global search

Microsoft's Graph RAG paper splits retrieval into two modes, and the distinction matters for how you design a system around it.

Local search — Starts from specific entities mentioned in the query and expands outward through their direct relationships. Good for questions about a particular entity: "What products does this company make?"

Global search — Uses the community summaries to answer questions about the dataset as a whole: "What are the main themes across all these documents?" Vector search can't do this at all, since no single chunk contains a dataset-wide view.

Where it helps and where it doesn't

Graph RAG isn't a universal upgrade over vector RAG. It's a different tool for a different shape of problem.

• Multi-hop reasoning across entities and relationships
• Questions that require a summary or theme across an entire corpus, not a single document
• Domains with a naturally graph-like structure: org charts, supply chains, citation networks, regulatory filings
• Answers where traceability to a specific relationship chain matters, since graph paths are auditable in a way that "top-k similar chunks" isn't

It costs more to build and maintain. Extracting entities and relationships accurately requires either a strong LLM pass over every document or a hand-tuned extraction pipeline, and errors compound: a missed or mislabeled edge quietly breaks every query that depends on it. For a simple FAQ bot or single-document Q&A, plain vector RAG remains faster to build and cheaper to run.

Rule of thumb: if your questions are "find the fact," use vector RAG. If they're "connect the facts," look at Graph RAG.

A few implementation notes

Hybrid retrieval — Most production systems don't pick one or the other. They combine vector search for initial candidate retrieval with graph traversal for relationship expansion, then merge both into the final context window.

Extraction quality is the bottleneck — The graph is only as good as the entity and relationship extraction step. Garbage extraction produces a graph full of duplicate entities and missing edges, and no amount of clever retrieval logic fixes that downstream.

Cost — Building the graph is a one-time (or periodic) batch cost, separate from query-time cost. Budget for re-running extraction as source documents get added or updated, not just for the initial build.