GraphRAG: Building Smarter Retrieval Workflows with Knowledge Graphs

Retrieval-augmented generation (RAG) is usually taught as one recipe: chunk your documents, embed the chunks, retrieve the nearest neighbours to a question, and let a language model answer from them. Cassie Shum's argument is that this recipe optimizes for local similarity and returns very little for global context, multi-hop reasoning, and provenance. Her correction is not a better retriever but a better substrate: a knowledge graph carrying business semantics, built next to the data an enterprise already owns, with agents on top as orchestration rather than as the intelligence.

Shum is VP of Field Engineering at RelationalAI, previously Head and Technical Director for Architecture and Development in North America at Thoughtworks. She gave this 50-minute talk at QCon AI New York 2025; InfoQ published the recording and transcript on July 1, 2026. She is unusually candid that the talk changed between abstract and delivery: when she proposed it GraphRAG's superiority looked decisive, and by the time she built the deck she conceded that rapid model progress had eroded part of that advantage. These notes report what she presented and mark where I add background.

What You Will Learn

• The four failure modes Shum attributes to vector-only RAG, and which of them larger context windows have partially fixed.
• What distinguishes a knowledge graph from a plain graph database, and why the difference is semantics rather than topology.
• How a GraphRAG pipeline extracts entities and relationships from unstructured documents and reconciles them into an existing ontology.
• Why she pushes business logic down out of the application layer into the semantic layer, and what that costs.
• The four agent capabilities she enumerates and the limit attached to each.
• Why "reflection needs ground truth" makes evaluation frameworks mandatory rather than optional.

Where Vector-Only RAG Breaks Down

The baseline workflow is familiar: embed the user's question, search a vector index for similar chunks, place the winners in the model's context window, and ask it to reason over them. Shum notes this was genuinely impressive on first contact, and that the cracks appeared quickly as query complexity and corpus size grew — which, in an enterprise, they always do. She groups the failures into four categories.

Aggravating all four is enterprise domain complexity: in finance, healthcare, and legal, data volume combines with dense, rule-laden domain logic that is hard to bring into one place at all.

Shum then marks her own claims to market. As of the talk she believes larger context windows have materially improved connecting the dots and multi-hop traversal, because you can feed a model ten documents — or a mix of documents, video, and images — and let it work across them. Global questions still fail, improving only in proportion to context supplied, since the model still reasons over a sample. Provenance is where she thinks vector RAG remains clearly deficient. That is the honest version of a vendor talk: the strongest surviving argument for GraphRAG is auditability, not raw answer quality.

What Makes a Knowledge Graph Different

Everyone reaches for nodes and edges on hearing "graph." Shum's distinction is that a knowledge graph adds semantics to that structure — the logic and business constraints connecting two entities, so the relationship itself carries meaning rather than mere adjacency. Her core principle follows from how language models behave: more context and more semantics yield more accurate decisions, so encoding meaning into edges is a way of supplying better raw material.

RelationalAI structures graphs in what Shum calls graph normal form, by analogy with first, second, and third normal form. The shape is long, narrow tables rather than wide ones — typically two entities and their relationship per row — which makes semantic traversal natural and lets reasoners run over the graph. Supplementary context, not from the talk: if you have modelled RDF triples or entity-attribute-value tables this will feel familiar, and the trade-off is the usual one — flexible schema evolution and easy traversal paid for with more joins and less obvious query plans. The layer giving the graph its meaning is the ontology: the formal description of which entity types exist and how they may relate.

Shum demonstrates with a deliberately mundane example, the Jaffle Shop model borrowed from dbt's starter kit — a food shop with products, stores, supplies, customers, items, and orders. She writes the graph in Python, creating orders with properties and relationships. The pointed example is a predicate on an order: is this a drink order or a food order? Historically that branch lived somewhere in the application layer. In the knowledge graph it lives on the data, in the semantic layer, which becomes the source of truth. That is the paradigm shift she argues for, illustrated with a trivial predicate precisely because the trivial case shows how far down logic can go. The payoff is that you can ask how two concepts relate and get an answer derived from modelled semantics rather than from documents scattered across Confluence and GitHub — valuable to product owners and the business, not only to engineers.

With only the structured tables loaded, Shum can already ask which customer ordered five times, which store had the lowest revenue, whether customers A and B shopped together. What she cannot ask is is customer X happy or why is this store struggling, because structured data does not contain that. The unstructured material — reviews, feedback, surveys — holds those answers, and folding it into the same graph is the job GraphRAG does.

Architecture And Data Flow

Shum's pipeline extracts entities and relationships from text to build or extend a knowledge graph, and she stresses it works in both directions: start from an existing graph and amend it, or start from documents and construct one. For the demo she ingested six PDFs of Jaffle Shop customer feedback, roughly 3,000 lines each — "not very big, but not that small" — framed as the equivalent of years of Yelp reviews. The pipeline pulled reviewers' names out as entities, the feedback itself as an entity, and created relationships of the form this customer gave this feedback at this store about this thing.

flowchart TD
    A[Unstructured documents
feedback, emails, surveys] --> B[Extract entities
and relationships]
    B --> C[Build hierarchy
and communities]
    C --> D{Reconcile against
existing ontology}
    D -->|new concept| E[Extend ontology with
logical expressions
and validations]
    D -->|matches existing| F[Amend knowledge graph]
    E --> F
    F -->|iterate on new documents| B
    F --> G[Extended knowledge graph]
    G --> H[Reasoners: SQL, similarity,
graph rules, prescriptive, predictive]
    H --> I[Agent orchestration layer]
    I --> J[Answer with provenance]

The loop is the part Shum emphasizes most. As documents arrive the pipeline revisits entities and reconciles them, asking whether a newly seen thing is an existing entity or a genuinely new one; seeing something repeatedly is evidence it deserves to become a concept. It then extends the ontology, extracting logical expressions and validations from the documents into the graph. The result answers questions neither the structured tables nor plain vector search could handle — not just "which store has the highest revenue" but "who was upset about that store in the last three months, why, and who recommended it." Her demo showed a query listing, for each of five problematic stores, the staff members named in negative feedback and which store had the most complained-about staff; another followed a ten-hop referral chain and identified which people in it had left negative feedback; a third ran Weakly Connected Components to discover friend groups from ordering patterns.

Her argument for persistence is the strongest practical case here. You could paste the same documents into ChatGPT, Gemini, or Claude and get something interesting, but you would not get history across millions of documents. The graph is the durable, growing artifact — state that accumulates rather than a one-shot context window, carrying history and provenance with it.

RelationalAI builds inside the Snowflake ecosystem, and Shum is explicit that she is not advocating Snowflake — Databricks, Google, and Amazon have equivalents. She is advocating the architectural property: build the graph on top of data where it already sits, inside an existing governance and security boundary, rather than exporting to a separate graph store such as Neo4j. She calls the export approach "not wrong," just different; her preference is to bring intelligence to the data. Pressed in Q&A on whether everything should therefore be consolidated, she backed away from the strong version: it depends where your data started, and you should not spend all your time centralizing just to make this work. Componentized workflows matter more — extract PDFs in a separate pipeline and bring the resulting graph in. Her Snowflake emphasis reflects a customer segment whose overriding requirement is don't move my data. On economics she reported Snowflake's separation of storage and compute: you pay for processing and query time, and storage is cheap enough that she described it as effectively unlimited — her characterization of a vendor's pricing posture, not an engineering guarantee.

Reasoners: What You Run Over the Graph

A reasoner is a capability invoked over the knowledge graph. Shum enumerates several families: query and similarity search, so vector retrieval becomes one tool among many rather than disappearing; rule reasoning over the business rules encoded in the edges, which is the capability a plain graph database lacks; graph algorithms such as PageRank and Weakly Connected Components, familiar from Neo4j; prescriptive reasoners, meaning solver-based optimization, her example being supply chain rerouting when a warehouse loses supply or a storm hits; and predictive reasoners such as graph neural networks that forecast future state from historical patterns. In RelationalAI's packaging these ship as a native app in the Snowflake marketplace, invoked against the graph with results written back to a Snowflake table. That detail is product-specific, but the layering is general: semantic layer and reasoners at the foundation, agents orchestrating above.

One Q&A exchange showed reasoners feeding back into modelling. Asked whether the pipeline can invent entity types never defined up front — an "influencer," say — Shum said yes, and that they use reasoners for it. Graph algorithms revealed customers connected to ten other customers, and that structural signal justified minting "influencer" as a new concept. The ontology is not purely hand-authored.

Agents: Four Capabilities, Four Limits

Shum's third act is the most transferable, because it applies whether or not you adopt knowledge graphs. She lists what agents can do, then bounds each capability.

Her summary is blunt: an agent is not a decision-making powerhouse, it is something you feed tools and directions to. The reflection limit carries the heaviest operational consequence — if reflection without ground truth merely launders errors, evaluation frameworks are not optional polish but the mechanism by which reflection has anything to reflect against. She returns to this repeatedly: ground-truth everything, and measure how accurate agents are as they iterate.

Brief definition: MCP, the Model Context Protocol, makes tools discoverable to an agent, which is why it appears in the planning row. In her demo the chat interface reached the graph over MCP, and the same tools could be wired to Claude or another agent. Notably the extraction step was not MCP — that was an LLM reading document text and turning it into nodes. Her inversion of the usual mental model is the thesis: agents are orchestration, and everything beneath them must be solid. Point every agent in the world at dirty data and you get nothing. She is explicit that knowledge graphs are how her team structures data and not the only valid answer; dbt-style transformation that cleans data counts as the same foundational work.

Trade-offs And Limitations

• The advantage is narrowing, and Shum says so. She wrote the abstract when GraphRAG's superiority looked clear and delivered the talk having concluded that six months of model progress closed part of the gap. Re-test any adoption decision against current models.
• GraphRAG is a tool, not a solution. This is her own closing framing; in the demo the agent chose it among several tools depending on context. The honest trigger is having unstructured data whose relationships and provenance matter.
• The strongest remaining case is provenance, not answer quality. Without a traceability requirement, and with mostly local questions, the cost of building and governing an ontology is harder to justify.
• Ontology construction is real work. The pipeline can extend an ontology and even mint concepts from graph algorithms, but the initial semantic model and its governance are engineering effort vector RAG does not require.
• Moving logic into the semantic layer relocates a maintenance burden. It becomes a queryable source of truth, but business logic now lives in a layer with its own deployment, testing, and review path that application developers must know to consult. Shum presents the upside; the operational cost is implicit rather than argued.
• Extraction quality bounds everything downstream. An LLM performs the extraction, errors propagate into the graph, and the reconciliation step deciding new-versus-existing entity is a heuristic, not a proof. This is precisely where ground-truth evaluation earns its place.
• The demo is a toy. Shum says so directly — six PDFs about a sandwich shop. Her experience is enterprise-scale but the artifacts shown are illustrative.
• Vendor context. Shum works for RelationalAI and the architecture shown is her product inside Snowflake. She repeatedly decouples principle from product, but the concrete tooling claims are about her own stack.

Practical Takeaways

• Classify your failing queries before adopting GraphRAG. If they fail on global context, multi-hop chains, or auditability, a graph substrate is plausible. If they fail on retrieval quality alone, fix chunking, embeddings, and reranking first.
• Test your current model with a larger context window before concluding vector RAG cannot connect the dots; Shum's own reassessment suggests part of the gap has closed.
• In regulated domains, treat provenance as a first-class architectural requirement rather than something to bolt on. It is the capability that survived her re-evaluation.
• Write the ontology explicitly. It is where business logic becomes queryable and where an agent can answer "how does this relate to that" from modelled semantics rather than scattered documentation.
• Design ingestion as an iterative loop from the start, so each new document reconciles against the existing graph. Compounding history is the differentiator against pasting documents into a chat window.
• Build the semantic layer next to the data inside its existing governance boundary, but do not centralize all data as a prerequisite; componentized pipelines let you extract elsewhere and bring the graph in.
• Do not treat agent memory as a correctness mechanism, and do not expect an agent to choose well between similar tools without explicit guidance.
• Build the evaluation and ground-truth framework before relying on reflection, which otherwise amplifies errors rather than correcting them.

Key Terms

• RAG (retrieval-augmented generation) — retrieving relevant information into a model's prompt so the answer is grounded in your data.
• Vector RAG — the common implementation: chunk, embed, retrieve by nearest-neighbour similarity to the embedded question.
• Knowledge graph — a graph whose edges carry semantics, business logic, and constraints, not merely connectivity.
• Ontology — the formal model of which entity types exist and how they may relate; the foundation a knowledge graph is built on.
• Graph normal form — RelationalAI's convention of long, narrow tables, typically two entities and one relationship per row.
• GraphRAG — extracting entities, relationships, and concepts from unstructured text to build or extend a knowledge graph used for retrieval.
• Multi-hop reasoning — answering a question requiring traversal of several chained relationships rather than one lookup.
• Provenance — the recorded path and sources behind an answer, supporting traceability and auditability.
• Reasoner — a capability run over the graph: SQL, similarity search, rule reasoning, graph algorithms, solver-based prescriptive optimization, or predictive models.
• Weakly Connected Components (WCC) — a graph algorithm partitioning a graph into groups where every node is reachable from every other ignoring edge direction; used in the demo to detect friend groups.
• MCP (Model Context Protocol) — a protocol exposing tools to an agent and making them discoverable.
• Ground truth — verified correct answers used to evaluate a system; the reference reflection needs in order to correct rather than compound errors.

Shum's closing message is deliberately unglamorous. Agents are not the problem solvers — you are, and agents make you more efficient at it. You still supply the logic, the tools, the memory, and the ground truth. The lasting value of the talk is less the specific pipeline than the layering it implies: foundations first, reasoners over those foundations, orchestration last, and a willingness to re-examine whether the technique you championed six months ago is still the right answer today.

Reference: Cassie Shum, Graph RAG: Building Smarter Retrieval Workflows with Knowledge Graphs, QCon AI New York 2025, published by InfoQ on July 1, 2026.