Overview

The first version answered questions.

That turned out not to be the same as earning trust.

It searched a generated knowledge base, selected a few matching chunks, and asked an LLM to compose a response. The pipeline worked, but the result had three recurring problems:

  1. Every answer followed the same report-like template, even when the question needed one sentence.
  2. A reranker could place polished documentation above newer implementation evidence.
  3. Citations proved that a document was retrieved, not that it supported the claim beside it.

The redesign changed the unit of quality from a fluent answer to an answer with inspectable evidence.

The Pipeline

question
  -> validate + classify
  -> retrieve candidate passages
  -> rerank for relevance
  -> re-apply source authority
  -> synthesize from numbered evidence
  -> validate citation mapping
  -> persist answer + evidence + feedback

The important detail is ordering. Relevance and authority are different signals.

A semantic reranker is good at answering, “Which passage sounds related?” It is not necessarily good at answering, “Which system is authoritative when two sources disagree?” Authority therefore runs after semantic reranking, and only for questions about current implementation.

People and process questions keep their normal relevance order. Implementation questions prefer source-controlled evidence over secondary documentation.

Separating Instructions from Evidence

The original prompt combined persona, rules, question, conversation history, and retrieved content into one user message. It worked, but the model treated every part as roughly equal.

The new prompt has two explicit layers:

  • System message — grounding rules, source precedence, safety constraints, tone, audience, and answer behavior.
  • User message — conversation context, the current question, and numbered source passages.

This also removed the fixed answer template. Structure is now adaptive:

  • A definition gets a short answer.
  • A comparison gets a table when a table helps.
  • An architecture question gets a staged explanation.
  • A troubleshooting question leads with the likely cause and next check.

Tone, audience, and verbosity are configurable, but grounding rules are not. A persona override may change how the answer sounds; it cannot change what counts as evidence.

Citation Integrity

The answer model sees sources as stable numbered blocks:

[Source 1: Service documentation]
<supporting passage>

[Source 2: Implementation notes]
<supporting passage>

Those numbers map one-to-one to the sources returned to the client. The mapping is created before synthesis and preserved through rendering.

The interface goes one step further: opening a citation shows the exact supporting passage, not merely a link to the parent page. That makes disagreement useful. A reader can see whether the source is outdated, incomplete, or genuinely contradictory.

Conversation as Durable State

Once answers became conversational, in-memory sessions were no longer enough. A restart should not erase the context behind a decision.

Conversation persistence added a few less-obvious requirements:

  • An interrupted stream must still close the turn cleanly.
  • Temporary client IDs must reconcile with persisted message IDs.
  • Deleted or failed conversations must disappear from the sidebar without leaving dead entries.
  • Feedback must remain attached to the answer version it evaluated.
  • Backups and health checks must expose persistence failures before users discover them.

The history interface is deliberately simple: a searchable, grouped list rather than a nested project tree. Retrieval complexity belongs behind the answer box, not in the navigation.

Rich Answers Without Unsafe Rendering

Technical answers benefit from tables and diagrams, but rendering model-generated markup creates a new boundary.

Markdown is parsed through a restricted renderer. Diagram blocks are handled separately with strict configuration, protocol checks, and a fallback to readable source text. A malformed diagram should degrade the answer, not break the page or create a navigation primitive.

The same principle applies to tables: make wide content responsive, keep the underlying text selectable, and never make presentation a requirement for understanding the answer.

Operational Feedback

Thumbs-up and thumbs-down buttons are only useful if they capture enough context to reproduce the problem.

Feedback records the answer, the selected evidence, and an optional reason. A diagnostic export can then package:

  • the recent conversation turns
  • the relevant backend log window
  • answer and citation metadata
  • a user-provided description of the failure

Logs are correlated by session and truncated from the oldest side, preserving the events closest to the reported problem. If a log file is unreadable, the export still succeeds with the remaining evidence.

The Next Architecture

The local system still uses an in-memory full-text index. The proposed production architecture keeps the same answer contract while changing the machinery underneath it:

source changes
  -> incremental ingestion
  -> normalize + chunk
  -> embed changed content only
  -> managed vector index + document store

question
  -> hybrid retrieval + reranking
  -> bounded live-tool fallback when evidence is thin
  -> grounded synthesis

The agentic portion is intentionally small. Deterministic retrieval handles normal questions. A bounded agent may fetch current documentation or structured facts only when confidence is low, the user explicitly asks for the latest state, or the answer depends on volatile data.

Agentic retrieval is a fallback, not the backbone.

What Worked

  • Treating authority as a separate ranking stage instead of another relevance weight.
  • Showing citation evidence rather than asking users to trust source titles.
  • Separating persona instructions from question and source content.
  • Keeping live tool use bounded and conditional.
  • Making diagnostics a product feature rather than a developer-only log hunt.

What Was Hard

  • Preserving citation numbering through reranking, token budgeting, and streaming.
  • Finalizing interrupted streams without creating half-written conversations.
  • Supporting rich Markdown while keeping model-generated content inert.
  • Distinguishing an outdated source from a poor retrieval result.

Takeaway

A knowledge assistant becomes useful when it can answer.

It becomes trustworthy when every important claim has a visible path back to evidence.