Vector similarity search is the default retrieval strategy for RAG systems. Embed your documents, embed your query, find the nearest neighbors. Simple and effective — until it isn’t.

The failure modes are predictable:

  • Semantic drift — Queries that are conceptually related but use different terminology miss relevant chunks
  • Missing relationships — Vector search treats each chunk independently, losing the connections between entities
  • Precision at scale — As the corpus grows, the nearest neighbors become increasingly noisy

Hybrid Retrieval Architecture

The pipeline I built combines three retrieval strategies:

  1. Dense vector search using ChromaDB embeddings for semantic similarity
  2. Sparse keyword search using BM25 for exact term matching
  3. Knowledge graph traversal using NetworkX for relationship-aware retrieval
results = []
results.extend(vector_retriever.retrieve(query, top_k=5))
results.extend(bm25_retriever.retrieve(query, top_k=5))
results.extend(kg_retriever.traverse(query, hops=2))

# Reciprocal rank fusion
merged = reciprocal_rank_fusion(results)

The Knowledge Graph Advantage

The real breakthrough came from the knowledge graph layer. By extracting entities and relationships from documents at ingestion time, the system can answer questions that require connecting information across multiple documents.

For example: “Which suppliers are affected by delays in Region X?” requires traversing supplier → product → region relationships that no amount of vector similarity will surface reliably.

Results

On our internal evaluation set:

  • Pure vector search: 67% relevance (top-5)
  • Vector + BM25: 74% relevance
  • Vector + BM25 + KG: 86% relevance

The knowledge graph layer added 12 percentage points of relevance, which translated directly to better downstream LLM responses.

Tradeoffs

The hybrid approach isn’t free. Knowledge graph construction at ingestion time adds latency and complexity. The graph needs maintenance as documents are updated or removed. And the fusion logic requires tuning per domain.

But for domains with rich entity relationships — supply chain, finance, healthcare — the investment pays off substantially.