Overview

Most “AI coding agents” stop at code generation. This one starts at the ticket and stops at the PR.

The platform is a five-service system that owns the round-trip. A story is drafted with you, decomposed into a plan, executed against the right repo, reviewed by a separate quality agent, validated against tests and coverage, and landed as a PR. The human approves the plan and steps in only on escalations.

The five services

Service Role Runtime
Story Agent Conversational ticket authoring, story generation, ideation FastAPI :8001
Dev Agent Headless poller + planner + executor Poller process + Dashboard :8011
Critic Agent Non-interventionist scoring gate before PR creation FastAPI :8020
Context Retriever Hybrid RAG over wiki, tickets, docs, and code FastAPI :8000
Frontend React SPA for chat, dashboards, execution monitor, search Static, served by story agent

Each service is independently deployable, and they communicate over HTTP.

Why no message queue

A queue would have made the architecture diagram neater. It would also have introduced new infra to operate, a new failure mode (lag, dead letters, redelivery), and a divergence point between ticket state and queue state.

The ticket system already supports what a queue would do here: ordered work, attachments, transitions, an audit trail. The poller polls, the critic is called synchronously, and the state lives in one place where humans can see it. Treating the ticket system as the state store removed an entire class of distributed-state bugs.

The dev agent

The dev agent isn’t a chatbot and doesn’t have a UI of its own. It’s a process that wakes up every few seconds, asks the ticket system “what’s the highest-priority ticket I can work on?”, and does the work: it plans the ticket, executes code in the right repo, sends the diff to the critic, applies feedback, and opens the PR.

Ticket selection is strictly priority-ordered, with the ticket closest to merging winning:

CODE REVIEW > PLAN APPROVED > IN DEVELOPMENT > IN TESTING > IN PLANNING > READY FOR PLANNING

This matters more than it looks. A ticket already in flight is never starved by a fresh one, and a PR-ready ticket gets dealt with first.

Once selected, the ticket flows through stage workflows:

READY FOR PLANNING  -> validate inputs (lane label, dev prompt, AC)  -> IN PLANNING
IN PLANNING         -> plan (v1) or refine (v2+), attach plan        -> PLAN APPROVED  (human approval)
PLAN APPROVED       -> derive branch, build prompt                   -> IN DEVELOPMENT
IN DEVELOPMENT      -> execute LLM in repo workspace, verify changes -> [critic gate]
[critic]            -> up to 3 attempts with feedback                -> IN TESTING
IN TESTING          -> build, test, coverage, static analysis        -> CODE REVIEW
CODE REVIEW         -> run repo tests, create PR                     -> poll PR checks

Every transition is idempotent. If the poller crashes mid-flow, the next cycle picks up where it stopped based on ticket state alone, so a crash never loses state.

Lane labels

Every ticket carries one of three labels: api, db, or ui. The label is the routing key for everything downstream: which repo to check out, which validation script to run, which branch naming convention to use, and which test runner to invoke.

api -> REPO_PATH_API,  pytest,       branch <EPIC>-api
db  -> REPO_PATH_DB,   pytest,       branch <EPIC>-db   (falls back to API)
ui  -> REPO_PATH_UI,   vitest/yarn,  branch <EPIC>-ui

This kept one platform serving three different stacks without per-language branching scattered through the code. Adding a new lane was a config block rather than a code change, and lane labels were the single most useful abstraction in the codebase.

Why the critic is a separate service

The original design had one agent that both wrote code and scored its own work. Scores were inflated. The fix was structural: separate the writer from the grader.

The dev agent owns the retry loop. It executes the prompt, captures output, sends the diff to the critic, and applies feedback. It never grades.

The critic agent owns the scoring. It only evaluates, never touches code, never commits, and never proposes fixes. It returns a structured result that says pass, fail with feedback, or escalate. Because the dev agent can’t lobby it, the critic stays honest: the agent that wrote the code never grades its own work.

The rubric

Four dimensions, fixed weights, blocking severity:

Dimension Range Weight Evaluates
Correctness 0-4 40% Logic, AC coverage, security
Code Quality 0-3 30% Patterns, error handling
Test Coverage 0-2 20% Unit + integration tests
Readability 0-1 10% Naming, structure

A review passes only if the total is at least 8 and every dimension clears its minimum floor. A blocker in the LLM’s output maps to a Python-enforced cap, so the LLM can’t feel generous past that cap. LLMs cannot be trusted to enforce their own thresholds.

Bias prevention

On retry, the previous critic review is passed back into the prompt, but with all numeric scores stripped. Any line starting with Score:, Correctness:, Code Quality:, and the like is removed before the prompt expands. This stops the LLM from anchoring on the prior score, so each attempt has to grade the actual diff rather than the last verdict.

Diff scoping

On shared epic branches, a naive working-tree diff includes other tickets’ commits sitting on the same branch, so the critic was being asked to grade code it had never seen. The fix is three-level filtering:

  1. Range: base_commit..HEAD (or origin/main...HEAD)
  2. Filter: git log --grep=({issue_key}), keeping only commits whose message tags this ticket
  3. Scope: the working-tree diff intersected with files touched by those commits

If scoping fails, whether from no commits found or no tag, the review is blocked rather than falling back to an unscoped diff. Failing closed beats reviewing the wrong code, and this one change ended the largest source of false-fail reviews. It took three iterations to get right.

Human in the loop, but cheap

The platform is autonomous, not unsupervised. Three explicit gates require a person:

  1. Plan approval — the implementation plan must be approved before code executes.
  2. Critic escalation — after repeated failed reviews, the ticket waits for human input.
  3. PR review — created PRs still go through normal review.

Between those gates, the agents run without nudging. The bet was that a cheap human in the loop, a one-click approval, beats an expensive one where you write the plan yourself.

A reviewer can also comment @critic on any ticket to force a fresh review, independent of the dev agent. The poller picks up the comment, syncs the repo, resets critic state, and runs a clean three-attempt loop. Review files use round numbering (_r2, _r3) so prior reviews aren’t overwritten. This became one of the most-used features: a human fixes the code by hand, drops the comment, and the critic re-grades.

When all attempts fail, the ticket transitions to escalated. A comment goes up with the full review trail, an email optionally fires to a configured list, and the PR gate stays closed until a human resolves it, either by fixing the code or overriding the gate. Escalation is the system saying this one needs you.

What worked

  • Treating the ticket system as the state store, which removed a class of distributed-state bugs.
  • Splitting the writer from the grader, keeping evaluation honest.
  • Lane labels as the routing primitive.
  • Python-enforced score caps.
  • The three-level diff scoping.
  • Idempotent stage workflows, so a poller crash never lost state.

What was hard

  • LLM CLI subprocess management: timeouts, kill signals, partial output buffering, prompt-boundary stripping, and token accounting from stderr and cost logging. These don’t matter at small scale and matter enormously at production scale.
  • Branch state policy. Allow dirty? Allow ahead? Allow behind? Each combination is a different failure surface, so it’s configurable with conservative defaults.
  • Auto-fix for test failures. Originally on, it was turned off when it started fixing the wrong tests, and it is now opt-in.
  • Observability for an LLM-driven system. Every LLM call now writes structured logs with cost, tokens, and a reasoning trace, otherwise the system is unfixable when it misbehaves.

The interesting work wasn’t the LLM calls. It was the scaffolding around them, the state machines, idempotent retries, scoped diffs, lane routing, and observability, plus the patience to make the human’s role one click instead of one hour. The agent owns the boring parts of the SDLC and lets the human approve the interesting ones.