Autonomous Dev Agent with a Separate Critic

Overview

The dev agent isn’t a chatbot. It doesn’t have a UI of its own. It’s a process that wakes up every few seconds, asks the ticket system “what’s the highest-priority ticket I can work on?”, and does the work.

Its job is to take a ticket through the SDLC pipeline without human babysitting — but with human checkpoints at the moments that matter: plan approval, critic escalation, PR review.

Why Split the Critic Out

The original design had one agent that both wrote code and scored its own work. Scores were inflated. The fix was structural: separate the writer from the grader.

The dev agent owns the retry loop. It executes the prompt, captures output, sends the diff to the critic, and applies feedback. It never grades.

The critic agent owns the scoring. It only evaluates. It never touches code, never commits, never proposes fixes. It returns a structured result that says pass, fail with feedback, or escalate.

This separation is what makes the critic useful. The dev agent can’t lobby it.

State Machine

The poller selects the next ticket by strict priority — closer to merging wins:

CODE REVIEW > PLAN APPROVED > IN DEVELOPMENT > IN TESTING > IN PLANNING > READY FOR PLANNING

This matters more than it looks. A ticket already in flight is never starved by a fresh one. A PR-ready ticket gets dealt with first.

Once selected, the ticket flows through stage workflows:

READY FOR PLANNING  → validate inputs                → IN PLANNING
IN PLANNING         → plan (v1) or refine (v2+)      → attach plan
PLAN APPROVED       → derive branch, build prompt    → IN DEVELOPMENT
IN DEVELOPMENT      → execute LLM in repo workspace  → critic gate
[critic]            → up to 3 attempts with feedback → IN TESTING
IN TESTING          → build, test, coverage, scan    → CODE REVIEW
CODE REVIEW         → run repo tests, create PR      → poll PR checks

Every transition is idempotent. If the poller crashes mid-flow, the next cycle picks up where it stopped based on ticket state alone.

Lane Labels

Every ticket carries one of three labels: api, db, or ui. The label is the routing key for everything:

api → REPO_PATH_API,  pytest,         branch <EPIC>-api
db  → REPO_PATH_DB,   pytest,         branch <EPIC>-db   (falls back to API)
ui  → REPO_PATH_UI,   vitest/yarn,    branch <EPIC>-ui

This kept the platform stack-agnostic. Adding a new lane was a config block, not a code change.

The Critic Rubric

Four dimensions, fixed weights, blocking severity:

Dimension	Range	Weight	Evaluates
Correctness	0–4	40%	Logic, AC coverage, security
Code Quality	0–3	30%	Patterns, error handling
Test Coverage	0–2	20%	Unit + integration tests
Readability	0–1	10%	Naming, structure

A review passes only if the total ≥ 8 and every dimension clears its minimum floor. A blocker in the LLM’s output is mapped to a Python-enforced cap — the LLM can’t “feel generous” past the cap.

Bias Prevention

On retry, the previous critic review is passed back into the prompt — but with all numeric scores stripped. Any line starting with Score:, Correctness:, Code Quality:, etc. is removed before the prompt expands.

This stops the LLM from anchoring on the prior score. Each attempt has to score the actual diff, not the last verdict.

Diff Scoping

On shared epic branches, the naive working-tree diff includes other tickets’ commits sitting on the same branch. The critic was being asked to grade code it had never seen.

The fix is three-level filtering:

1. Range: base_commit..HEAD (or origin/main...HEAD)
2. Filter: git log --grep=({issue_key}) — only commits whose message tags this ticket
3. Scope: working-tree diff intersected with files touched by those commits

If scoping fails — no commits found, no tag — the review is blocked rather than silently falling back to an unscoped diff. Failing closed beats reviewing the wrong code.

Human-Triggered Re-Review

A reviewer can comment @critic on any ticket to force a fresh review, independent of the dev agent. The poller picks up the comment, syncs the repo, resets critic state, and runs a clean 3-attempt loop. Review files use round numbering (_r2, _r3) so prior reviews aren’t overwritten.

This turned out to be one of the most-used features. A human fixes the code by hand, drops the comment, the critic re-grades.

Escalation

When all attempts fail, the ticket transitions to escalated. A comment goes up with the full review trail. Optionally, an email fires to a configured list. The PR gate stays closed until a human resolves it — either by fixing the code or by overriding the gate.

Escalation isn’t a failure mode. It’s the system saying this one needs you.

What Worked

• Splitting the writer from the grader.
• Lane labels as the routing primitive.
• Python-enforced score caps. LLMs cannot be trusted to enforce their own thresholds.
• The 3-level diff scoping. One change ended the largest source of false-fail reviews.

What Was Hard

• Subprocess management across LLM CLIs — timeouts, kill signals, partial output buffering, token accounting from stderr.
• Branch state policy. Allow dirty? Allow ahead? Allow behind? Each combination is a different failure surface. Configurable, defaults conservative.
• Auto-fix for test failures. Originally on; turned off when it started fixing the wrong tests. Now opt-in.

Takeaway

The agent isn’t valuable because it writes code. It’s valuable because it owns the boring parts of the SDLC and lets the human approve the interesting ones.