Agentic RAG with Iterative Refinement

Overview

Most RAG systems fail in three predictable ways:

1. The retrieved chunk is related to the query but missing the piece that actually answers it.
2. The retrieved chunk is the wrong entity entirely but scores high enough to look right.
3. The chunk is correct but references something it doesn’t include — a constant, a config file, a parent class.

Throwing more chunks at the problem doesn’t fix it — it dilutes precision. The fix was to put the LLM inside the retrieval loop, not just at the end.

Architecture

Query
  → [Intent Extraction]            LLM: classify query, extract entities, suggest sources
  → [Structural Filter Extraction] regex: file paths, entity IDs, repo names
  → [Hybrid Search]                semantic (pgvector) + keyword (FTS) + graph edges
  → [LLM Evaluation]               scores relevance + completeness, names the gap
       ├── relevant + complete   → finalize
       ├── relevant + incomplete → expand context (parent / neighbors / config)
       └── irrelevant            → refine search (drop filters, regen query)
  → [Final Answer + Reasoning Trace]

The agent iterates up to N times. Most queries finish in one pass; the hard ones use two.

The Three Retrievers

Retriever	What It Catches
Dense vector (pgvector + 384-d sentence-transformer)	Semantic similarity, paraphrase, intent
Sparse keyword (Postgres FTS)	Exact identifiers, class names, error codes
Graph traversal (entity edges)	Multi-hop relationships (calls, imports, references)

Scores combine as:

hybrid_score = (semantic_weight * semantic_score) + (keyword_weight * keyword_score)

For code queries, semantic weight is boosted to 0.85 since code embeddings carry meaning that FTS misses. For ticket queries, the default 0.7 / 0.3 split holds.

Iterative Refinement: The Decision Matrix

Relevance	Completeness	Action
> 0.7	> 0.8	finalize
> 0.6	< 0.7	expand_context
< 0.5	any	continue_search

When the LLM picks expand_context, it also returns a structured expansion strategy:

{
  "fetch_parent_entities": true,
  "fetch_related_entities": false,
  "fetch_config_files": false,
  "entities_to_expand": ["EmailService.java"]
}

That structured output is what makes the loop deterministic. The agent isn’t guessing what to do — the LLM tells it, in JSON.

Worked Example

A user asks for a constant referenced inside a function. The function chunk shows the reference but not the value.

Iteration 1
  Found: EmailService.java::sendEmail  (lines 45–62)
  Relevance: 0.85   Completeness: 0.45
  Quality: relevant_incomplete
  Gap: "Function references DEFAULT_HOST but constant not shown"
  Strategy: { fetch_parent_entities: true }

Iteration 2
  Fetched: EmailService.java  (full class, lines 1–121)
  Relevance: 0.95   Completeness: 0.90
  Quality: relevant_complete
  → finalize

The final answer shows the actual constant value, because the parent class came in on iteration 2.

Intent Extraction

Before any search runs, an LLM classifies the query into one of 11 intent categories and extracts structured entities. The output drives source filtering:

Intent	Sources Searched
code_search	Code repos only
issue_tracking	Ticket system only
documentation	Docs only
architecture	Docs + code

Caller context narrows it further: a “dev agent” caller only sees code; a “story agent” caller only sees docs and tickets. Source filtering cut search time 40–60% and removed whole classes of false positives — the “documentation” hit when you wanted code.

Post-Retrieval Pipeline

Five stages between raw retrieval and the LLM:

1. Deduplication — exact ID match, parent grouping, 95% semantic overlap
2. Score filtering — drop below 0.3, dynamic percentile thresholds
3. Staleness check — flag chunks older than 365 days
4. Re-ranking — composite: 50% original, 20% recency, 15% source priority, 15% metadata
5. Token budget — 4000-token cap, round-robin across source groups for balanced representation

Results

Configuration	Relevance@5	Relevance@10
Vector only	67%	72%
+ Keyword (FTS)	74%	79%
+ Graph traversal	86%	89%
+ Iterative refinement (complex queries)	85% (from 25%)	—

Latency tradeoff: +1–6 seconds for complex queries in exchange for the relevance gain. Simple queries are unchanged.

Observability

Every LLM decision is logged with a tag:

[LLM_STATE]     Iteration 1: Relevance=0.85, Completeness=0.45, Quality=relevant_incomplete
[GAP_ANALYSIS]  Function references DEFAULT_HOST but constant not shown
[FETCH_PARENTS] Added parent: EmailService.java
[LLM_STATE]     Iteration 2: Relevance=0.95, Completeness=0.90, Quality=relevant_complete

The same trace is returned in the API response. Nothing about the iteration is opaque — debugging the agent never requires guessing what it was thinking.

Takeaway

The unlock wasn’t a better embedding model. It was making the LLM responsible for noticing when retrieval almost worked — and saying, in structured output, what was missing.