Knowledge graphs and GraphRAG

Vector similarity answers “what is semantically close to this query” — it has no native notion of structured relationships between entities. A knowledge graph explicitly represents entities as nodes and typed relationships as edges (Company --acquired--> Company, Person --works_at--> Company), which lets you answer questions vector search cannot: multi-hop queries (“who is the CEO of the company that acquired the company Jane founded” — a chain of edges, not a single semantic match), aggregation over relationships (“how many companies did this fund invest in”), and precise, non-fuzzy relationship queries where semantic similarity would be too imprecise (you want the exact reports_to edge, not “text that resembles a reporting relationship”). The trade-off is that a graph requires the relationships to be extracted and correct in the first place — vector search degrades gracefully with messy data, a wrong or missing edge in a graph silently breaks any query that depends on it.

An LLM (or a dedicated NER/relation-extraction model) reads each document/chunk and extracts (entity, relation, entity) triples, which are then resolved and merged: the same real-world entity mentioned differently across documents (“Acme Corp”, “Acme”, “Acme Corporation”) must be recognized as one node (entity resolution / coreference) or the graph fragments into duplicate nodes that never connect. Common failure points: extraction errors (the LLM invents a relation that is not actually stated, or misses one that is); entity resolution failures — under-merging leaves duplicates and breaks multi-hop traversal, over-merging conflates genuinely distinct entities (two different "John Smith"s); relation drift over time — an edge true when extracted (a person’s job title) becomes stale as reality changes, and graphs are not automatically freshness-aware the way a simple re-embed-on-update pipeline is; and schema rigidity vs. real-world messiness — deciding the relation vocabulary up front (works_at vs employed_by vs member_of) versus letting it be open and dealing with the resulting sprawl.

Standard chunk-based RAG retrieves individual passages by similarity, which works for questions answerable from one or a few localized chunks but struggles with global, corpus-wide questions (“what are the main themes across all these reports” — no single chunk contains “the main themes”; that requires synthesizing across the whole corpus). GraphRAG builds a knowledge graph over the corpus, clusters it into communities of related entities, and pre-generates summaries at each level of the community hierarchy (a summary of a small tight cluster, a summary of a larger super-cluster, up to a whole-corpus summary). A global query is then answered by retrieving and combining the relevant community summaries rather than searching for individual matching chunks — because the summarization work was done up front at indexing time, the query-time step gets pre-digested, corpus-level context that no single-chunk vector search could produce. The real cost: significant indexing-time compute (LLM calls to extract entities/relations and generate summaries at every hierarchy level) and higher operational complexity than a vector index, which is why it is reached for specifically when the query distribution genuinely includes global/thematic questions, not as a default upgrade over vector RAG.

• Exact multi-hop relationship questions — “which suppliers does this company’s largest customer also use” is a graph traversal (two or three hops along explicit edges); a vector search over document chunks has no mechanism to compose that chain reliably.
• Provenance and explainability requirements — a graph traversal’s path is the explanation (“here is the chain of edges that produced this answer”), whereas a vector match’s relevance is a similarity score with no structural justification.
• Highly structured domains with a natural graph shape — org charts, supply chains, citation networks, regulatory dependency chains — where the relationships, not the prose describing them, are the actual information need.
• Aggregation and counting over relationships — “how many direct reports does each VP have” is a graph query; it is not something semantic similarity over text answers reliably at all.

Vector search remains better for open-ended, fuzzy, “find text about this topic” queries where no precise relationship structure is being asked for — most retrieval-augmented Q&A over prose falls into this category, which is why GraphRAG is a complement to vector RAG for a specific slice of queries, not a wholesale replacement.

• Build cost — extraction and entity resolution over a large corpus is expensive (many LLM calls) and slow compared to embedding chunks.
• Maintenance cost — keeping a graph accurate as source documents change requires re-extracting and re-resolving affected entities/edges, which is a harder incremental-update problem than re-embedding a changed chunk; naive full rebuilds do not scale.
• Query complexity — graph query languages (Cypher, Gremlin, SPARQL) and multi-hop traversal planning are a genuinely different skill set from configuring an ANN index, and query performance can degrade badly on poorly modeled or overly dense graphs (hub nodes with huge fan-out slow traversal).
• Quality is data-quality-bound — a graph is only as good as its extraction and resolution accuracy, and errors compound silently across multi-hop queries in a way a single bad chunk in vector RAG does not (one bad chunk affects one retrieval; one bad edge can corrupt every query path that traverses it).

Reach for a graph only once you have a concrete, recurring query pattern that vector (or hybrid lexical+vector) search demonstrably cannot serve — building one speculatively is a common source of wasted effort.

A common pattern: use vector or hybrid lexical+vector search to find an initial set of relevant entities or documents (the entry points into the graph), then traverse the graph from those entry points to pull in structurally related context the initial similarity search would have missed (related entities, upstream/downstream relationships), and optionally re-rank or fuse the combined candidate set before it reaches the generator. This plays to each method’s strength: vector search is good at “find things about this topic” from free-form text, graph traversal is good at “now follow the precise relationships from there.” The engineering cost is real — you are now maintaining and keeping in sync a vector index, possibly a lexical index, and a graph store, each with its own freshness and consistency story — so this combination is reserved for query distributions that genuinely need both “fuzzy topical relevance” and “precise relational reasoning” in the same system, not adopted as a default architecture.