Deep dive · August 17, 2026 · 12 min read
Neo4j vs vector databases: which one (and when to use both)
A practical, engineer first comparison for teams building enterprise LLM applications. When to reach for a graph, when for a vector store, and why the right answer for most production systems is a hybrid.
TL;DR
- Neo4j is a graph database. It excels at storing typed relationships between entities and traversing them at query time. Use it when your data is a network and your questions involve relationships.
- Vector databases (Pinecone, Weaviate, pgvector, Milvus) store and retrieve dense embeddings. They excel at semantic similarity over unstructured text. Use them when your data is a large pile of documents.
- Hybrids win in production. Most non trivial LLM applications end up using both: a knowledge graph for entities and structured facts, a vector index for supporting text. GraphRAG is the name for this pattern.
What each technology is actually for
Neo4j is a native graph database. Data is modelled as nodes (entities) connected by edges (relationships), both of which can carry properties. You query it in Cypher (or GraphQL, or Gremlin) with expressions that traverse the graph: MATCH (c:Customer)-[:HOLDS]->(a:Account)-[:TRANSACTED_WITH]->(cp:Counterparty) WHERE cp.risk_score > 0.9 RETURN c. The engine is optimised for these traversals: you can walk five hops in a graph faster than you can do the equivalent JOIN in a relational database over the same data.
A vector database stores high dimensional embeddings: the numeric representation of a chunk of text (or image, audio, code) produced by a model. You query it by embedding a search string and asking for the k nearest neighbours in vector space. This gives you results that are semantically similar to the query, even when no keywords overlap. That is powerful for search and for RAG, but it has no notion of "relationship" between the chunks it returns.
The comparison at a glance
| If this is true for you… | Neo4j | Vector DB |
|---|---|---|
| Your primary retrieval unit is | Typed entities and their relationships | Chunks of unstructured text |
| You need multi hop reasoning | Native (Cypher/GraphQL/SPARQL traversal) | Not supported: chunks are independent |
| You need semantic search over prose | Possible with add ons, not the strength | Core capability |
| You need fact level provenance | First class: every edge points to its source | Chunk level at best |
| Your data model is a network (fraud, KYC, org charts, supply chains) | Obvious fit | Poor fit: no relationship model |
| Your data is a large pile of documents with no meaningful relationships | Overkill | Obvious fit |
| You need to answer 'which X are connected to Y via Z' | One query | Not possible: no notion of connection |
| You need to answer 'find text similar to this passage' | Possible but awkward | One query |
Where teams pick wrong (and why)
The most common failure mode we see in enterprise RAG pilots is using a vector database for questions that are fundamentally about relationships. "Which of our vendors supply components used in products currently under recall?" is not a similarity question: it is a four hop traversal (Vendor → Component → Product → Recall). Chunking the vendor catalogue and stuffing it into Pinecone cannot answer it, no matter how well you tune your embedding model.
The other common failure mode is the reverse: standing up Neo4j for a workload that is 95% semantic search over policy documents, product manuals, or support transcripts. A vector database with good chunking, metadata filters, and a re ranker will beat a graph here every time.
The right question is not "which one wins": it is "which of my questions are relationship shaped and which are similarity shaped?". Most real systems have both.
The hybrid pattern (GraphRAG)
The production pattern most enterprise teams converge on looks like this:
- Structured facts and relationships → knowledge graph. Customers, contracts, transactions, org charts, product hierarchies. Ingested from systems of record via CDC.
- Unstructured text → vector index. PDFs, transcripts, tickets, wiki pages. Chunked, embedded, and linked back to the entities they reference.
- Both indexed by the same entity IDs. A retriever can start in the graph, find the relevant customers or contracts, and hop into the vector index to fetch the supporting text: or vice versa.
- LLM sees a grounded context bundle. Structured facts (from the graph) plus supporting quotes (from the vector store), with citations back to both. Hallucination rate drops. Multi hop questions become answerable. Answers cite their sources.
We covered the architecture in more depth on the GraphRAG implementation page: including the reference architecture, delivery phases, and how we measure the before/after.
A decision heuristic
When someone asks "should we use Neo4j or a vector database?", we walk them through three questions:
- Is the answer to your top 3 user questions in a single document, or spread across multiple entities? Single document → vector wins. Multiple entities → graph wins.
- Do your users need citations at the fact level or the document level? Fact level → graph. Document level → vector.
- What is your current hallucination rate? If it is above 5% on questions that require reasoning across systems, no amount of vector tuning will fix it. You need a graph in the retrieval path.
If two of these three point to graph, budget for a hybrid stack and start with a small target question set. If all three point to vector, you probably do not need a graph yet.