Overview

A chat box over a ticket system is easy to mock up. It’s the second message that breaks it.

Mock: “show me TKT-123.” Easy.

Reality: “show me the description, then create three subtasks under it with the AC I just listed, then assign them to the people I mentioned earlier.”

That’s not one LLM call. That’s planning, state, recovery, and verification.

Three Paths, Picked Automatically

The chat backend has three execution paths. The routing happens up front, based on what the LLM extracts from the message.

Path 1: Single-step pipeline

Atomic action, no dependencies. show TKT-123. list sprints. change priority of TKT-45 to P1.

intent extraction → scope guard → resolve missing fields → execute tool → render

Fast. Deterministic. One round-trip.

Path 2: Multi-step agent loop

Anything that requires sequencing or cross-step references.

LLM decomposes into ≤ 10 ordered steps
  → preview as PlanCard
  → user approves
  → sequential execution with $stepN.result.field resolution
  → on failure: retry / replan / dynamic adjust
  → after completion: goal verification

Steps can reference earlier outputs. The plan card shows every step before any of them run. The human approves once, the whole plan executes.

Path 3: Conversation agent

LLM-driven. Full conversation history goes in, the model decides per turn whether to respond conversationally or call a tool. Tool results feed back into the model up to N iterations until it produces text.

This is what handles “actually, change the priority on the second one to P1” — the model has the context to know which one “the second one” was.

Why Plans Need Approval

Letting an LLM execute a multi-step plan without preview is one of those decisions you only make once.

The PlanCard renders every step before anything runs. Two reasons:

  1. Safety. A wrong plan is easier to catch than a wrong commit.
  2. Trust. Users approve quickly once they trust the planner. Until then, they don’t approve at all.

The cost of the preview is roughly zero (one extra render). The cost of skipping it is everything.

Error Recovery

A step fails. Three things can happen:

  1. Auto-retry — transient errors (timeouts, 429s) get exponential backoff
  2. Replan — non-transient failure triggers the LLM to propose an alternative for that step
  3. Dynamic adjust — a step succeeds but its result invalidates downstream steps (e.g., the search returned no matches that the next step was supposed to operate on); the agent re-evaluates remaining steps

The classifier deciding which path applies is small but load-bearing. Misclassify a transient failure as permanent and you waste an LLM call replanning a deploy hiccup. Misclassify a permanent one as transient and you retry into the same wall.

Goal Verification

After a plan completes, the agent re-reads the original goal and asks: did we actually do it?

If yes → return. If no → execute follow-up steps.

This catches the case where every step “succeeded” but the goal wasn’t met. The plan can be technically correct (each tool returned 200) and substantively wrong (the AC field is still empty because the LLM filled the description instead).

Wiki-First Context

Every grounded response checks the project wiki before reaching out to the external retriever.

The wiki is a directory of markdown entities with YAML frontmatter — Components, Features, Dependencies, Decisions, Runbooks, Glossaries. Each one is hand-curated.

The order is deliberate:

  1. Wiki — hand-curated, project-specific, low latency
  2. Context retriever — broad, indexed, useful as fallback
  3. Nothing — proceed without external context if both fail

When the wiki has a hit, the retriever isn’t called. This kept LLM grounding both accurate (the wiki is true by construction) and cheap (a local file read beats a network call every time).

Tool Surface

19 active tools in the registry, covering view / list / search / create / update / clone tickets, bulk operations, story generation from epics, design analysis, story ideation with wiki-backed recommendations, summarization, duplicate detection, priority suggestion, and AC generation.

Each tool has a Pydantic input schema and a unified result contract. The LLM doesn’t see free-form text from tools — it sees structured results normalized through one type.

Frontend Cards Matter

The chat is not just text. Specific tool results render as cards:

  • DraftIssueCard — editable create/update drafts with per-field AI refinement
  • PlanCard — multi-step plan preview with approve / cancel
  • ShowIssueCard — full ticket detail
  • KnowledgeResultCard — RAG result with confidence and source badges
  • DesignPreviewCard — story classification + code context
  • WikiRecommendationsCard — wiki-backed architecture suggestions

The cards make the conversation operate-able instead of just readable. You don’t retype the description to edit it — you click into the card.

What Worked

  • The three-path routing. Most messages are single-step; not running them through the planner is a 10x latency win.
  • Plan previews. Saved more bad outcomes than any other single feature.
  • Structured tool I/O. The agent never parses free-form LLM tool output.
  • Wiki-first grounding. Curated content beats indexed content when accuracy matters.

What Was Hard

  • The conversation agent’s iteration cap. Too low and it stops mid-thought; too high and a runaway tool loop burns tokens. Settled at 10 with a hard timeout.
  • Slimming tool results before re-prompting. Raw tickets are huge; passing them whole into the LLM context blew the budget. The slimmer preserves the fields most likely to matter.
  • Sanitizing LLM CLI output. CLIs occasionally leak metadata or partial prompts. Output is sanitized before it reaches synthesis.

Takeaway

The interesting thing isn’t that an LLM can call an API. It’s that the LLM shouldn’t be the one deciding when to call it.

The planner decides. The user approves. The LLM executes. The verifier checks. Each role is small, separate, and replaceable.