Overview
A chat box over a ticket system is quick to mock up. The second message is what breaks it.
The mock handles “show me TKT-123.” Reality asks for “show me the description, then create three subtasks under it with the AC I just listed, then assign them to the people I mentioned earlier.” That is not one LLM call. It is planning, state, recovery, and verification.
The first version of the assistant handled this by planning before it acted. The version I run now keeps the same safety boundaries but moves where planning lives. This is the story of why that change happened and what it cost.
The first design: plan before acting
The original backend had three execution paths, routed up front based on what the LLM extracted from the message.
The single-step pipeline covered atomic actions with no dependencies: show TKT-123, list sprints, change priority of TKT-45 to P1. Intent extraction fed a scope guard, missing fields were resolved, the tool executed, the result rendered. One round-trip, deterministic.
The multi-step agent loop handled anything that required sequencing or cross-step references. The LLM decomposed the goal into at most ten ordered steps and previewed them as a PlanCard. The user approved once, then the whole plan executed sequentially, resolving $stepN.result.field references between steps so a later action could operate on an earlier output.
The conversation agent was LLM-driven. Full history went in, and the model decided per turn whether to answer or call a tool, feeding tool results back until it produced text. This path handled “actually, change the priority on the second one to P1,” because it had the context to know which one “the second one” was.
Plan approval was the load-bearing decision. The PlanCard rendered every step before anything ran, for two reasons. A wrong plan is easier to catch than a wrong commit, and users approve quickly once they trust the planner but not at all before then. The preview cost roughly one extra render.
Error recovery lived in the loop. Transient errors like timeouts and 429s got exponential backoff. A non-transient failure triggered the LLM to propose an alternative for that step. And when a step succeeded but its result invalidated downstream steps, for instance a search returning no matches for the next step to operate on, the agent re-evaluated the remaining steps. The classifier deciding which case applied was small but consequential: misread a transient failure as permanent and you waste a call replanning a hiccup; misread a permanent one as transient and you retry into the same wall.
After a plan completed, the agent re-read the original goal and asked whether the work was actually done. This caught the case where every tool returned 200 but the goal was still unmet, such as the AC field staying empty because the LLM filled the description instead.
Underneath all of this sat a registry of typed tools covering view, list, search, create, update, and clone operations, bulk actions, story generation, design analysis, summarization, and AC generation. Each had a Pydantic input schema and a unified result contract, so the agent never parsed free-form text from a tool. Specific results rendered as cards rather than prose: editable drafts with per-field refinement, plan previews, full ticket detail, knowledge results with source badges. The cards made the conversation operable instead of merely readable.
Why it needed to change
The first design was safe, and for genuinely risky work it was right. But it made ordinary conversation feel like workflow software.
Two problems compounded. First, the three paths drifted. The same design-analysis action could produce different response shapes depending on whether it arrived as free text, a suggestion button, or an approved plan, so fixes had to be repeated across branches. Second, the plan card imposed the same ceremony on very different risks. Creating several dependent work items deserved review. Reading a sprint and then listing its items did not, yet both paid the same tax.
The goal of the rewrite was never more autonomy. It was fewer competing abstractions.
The second design: two paths, one executor
The current architecture has two entry paths that converge.
free-text message
-> conversation agent
-> choose tool or answer
-> execute tool
-> observe result
-> repeat within hard limit
shortcut / suggestion / missing-field reply
-> deterministic resolver
-> execute tool
both
-> shared result contract
-> shared card predicate
-> shared renderer
The conversation agent receives full history and may chain tools within a bounded loop, but it cannot invent capabilities. Every action still passes through typed schemas, permission checks, and the same executor the deterministic UI flows use. That convergence fixed more than duplicated code. Once a tool returns a result, presentation no longer depends on how the request entered the system, which makes behavior explainable.
Splitting the model work helped too. Choosing a tool is classification; explaining a result is synthesis. Treating both as one LLM task wasted latency and reasoning budget, so dispatch runs on a lower reasoning tier and a stronger tier engages only when a user-visible response needs composition. Many read operations skip synthesis entirely and pass structured results through deterministic formatters, producing stable links, chips, and tables without another model call.
Design planning as a selectable contract
Design generation moved away from a single opaque prompt. The assistant first analyzes a work item and returns two selectable lists: considerations the design should address, and sections the document should contain. The user can drop irrelevant concerns or add missing sections before generation, and the resulting document always ends with explicit key decisions.
For a batch of related work, those decisions become context for the next item, so a later design does not reopen the caching, locking, or API choices an earlier one already settled. This is a compact list of decisions with downstream consequences rather than a replayed chat transcript.
Independent story criticism
Generated work items are reviewed by a separate critic across dimensions like completeness, clarity, acceptance criteria, scope, persona alignment, and decomposition. The author does not grade itself.
When revision is needed, the user sees dimension-level suggestions and picks which to apply. Approved dimensions are rewritten surgically; rejected ones leave the original intact. Repeated failure escalates with the review history rather than cycling silently. Sibling work is included in the review context, so a UI story is not penalized for behavior an API story deliberately owns. The critic is a gate with human control, not an automatic rewrite loop.
UI-first prototyping
UI work takes a different path before design generation. The assistant selects the closest screen from a regularly refreshed snapshot catalog of the real application, combines that template with route-specific sample data, and asks the LLM to apply only the requested change. The prototype then runs through a browser smoke test: labels, filters, dialogs, and primary interactions must work, not merely render. The design discussion starts from the current product rather than an LLM’s memory of what enterprise software tends to look like.
Reliability lessons
Several smaller changes mattered as much as the architectural rewrite. Message count and total content are capped before prompts are assembled. Coding-agent CLI output is sanitized before it reaches user-visible synthesis, since CLIs occasionally leak metadata or partial prompts. Expensive analysis is cached, but mutable work-item data is refetched before generation so cached analysis never becomes stale input. Critic availability is an explicit state, because “unknown” is not the same as “disabled.” Confirmation flags are preserved across scope and planning transformations, and in-memory design and conversation state is bounded with eviction.
Agent systems rarely fail at the center of the happy path. They fail in the seams between state representations.
What the arc taught me
The first design was correct about the danger of letting an LLM decide when to act, and plan previews saved more bad outcomes than any other single feature. What it got wrong was applying that ceremony uniformly. The rewrite kept the boundaries that earned their keep, the typed tools, the permission checks, the human approval on risky work, and removed the ones that only added friction. The hardest part was removing the legacy planner without changing visible behavior, and preserving context across chained tool calls without passing oversized raw results back to the model. The system does more reasoning now, but it is easier to reason about, because every path shares the same tools, contracts, and boundaries and the model only reasons where the request is genuinely ambiguous.