Testing Non-Deterministic Agents in CI

The Problem

Traditional CI asks deterministic questions:

Did the function return the expected value?
Does the response match the schema?
Did the test suite exit successfully?

Agent outputs are different. Two valid plans may use different words, order their sections differently, or choose different but equivalent tool sequences.

Exact snapshots become brittle. Removing assertions entirely is worse.

The useful question is not, “Can an LLM judge another LLM?” It can.

The useful question is, “Which decisions are safe enough to delegate to that judge?”

Start with Three Layers

No single evaluation technique should own the gate.

Layer	Method	Best for
Contract	Schema, regex, exact invariants	Required fields, valid tool names, forbidden content
Semantic	Embedding similarity, reference sets	Meaning-preserving variation
Behavioral	Rubric-based LLM judge	Completeness, reasoning quality, persona, plan adherence

Run them in that order.

A missing required field should fail before an expensive judge call. A prompt-injection leak should not receive a nuanced quality score. Deterministic failures are cheaper, clearer, and easier to reproduce.

The LLM judge handles only the residue: qualities that matter but cannot be expressed reliably as exact assertions.

DeepEval as the Test Harness

DeepEval fits naturally into a Python test suite. Each agent run becomes a test case with explicit inputs, actual output, and—where available—expected context.

from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

plan_quality = GEval(
    name="plan_quality",
    criteria=(
        "The plan covers every acceptance criterion, references only supplied "
        "components, explains risky changes, and includes a verification step."
    ),
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.8,
)

case = LLMTestCase(
    input=story_fixture,
    actual_output=generated_plan,
)

assert_test(case, [plan_quality])

The important part is not the library call. It is the rubric.

“Is this a good plan?” is not a test. A useful rubric names observable properties, available evidence, forbidden assumptions, and the threshold for failure.

Promptfoo for Prompt-Level Regression

DeepEval works well inside the application test suite. Promptfoo is useful one level earlier: comparing prompt or model changes across a fixed matrix of cases.

prompts:
  - file://prompts/planner-v1.txt
  - file://prompts/planner-v2.txt

tests:
  - vars:
      story: file://fixtures/api-change.json
    assert:
      - type: contains
        value: "Verification"
      - type: not-contains
        value: "frontend/"
      - type: llm-rubric
        value: "The plan stays within the API layer and covers every criterion."

This makes prompt changes reviewable as experiments rather than intuition. The same fixtures run against both versions, and the result can be stored as a CI artifact.

Use Promptfoo to compare behavior. Use the normal test runner to enforce application contracts. They overlap, but they are not substitutes.

Why One Judge Is Not Enough

An LLM judge has its own preferences:

verbosity bias
position bias
preference for familiar phrasing
inconsistent treatment of borderline cases
drift when the underlying model changes

A dual-judge design reduces—but does not eliminate—those risks.

candidate output
  -> primary judge
  -> secondary judge from a different model family

agreement above confidence margin
  -> verdict

material disagreement
  -> stronger adjudicator or human review

The second judge should not be a differently named deployment of the same underlying model. Model-family diversity is the point.

Running two judges on every test can be expensive, so use it selectively:

every safety-critical case
every borderline primary score
a rotating sample of normal cases
the full suite on a scheduled run

Calibration Before Enforcement

Do not turn a rubric into a hard merge gate because it looked reasonable in five examples.

Build a human-labeled benchmark set containing:

clearly good outputs
clearly bad outputs
plausible but incomplete outputs
adversarial prompt-injection attempts
equivalent answers with different wording and structure

Measure judge agreement with humans, false positives, false negatives, and run-to-run flips.

Roll out behavioral checks in stages:

Observe — record scores without affecting CI.
Warn — annotate pull requests when the judge fails.
Gate narrowly — block only calibrated, high-confidence rubrics.
Expand — promote additional checks after they meet the reliability target.

Calibration is not a one-time launch task. Re-run it when prompts, models, or product expectations change.

Failure Policy

Judge infrastructure must not hold a repository hostage.

A practical fallback chain is:

Retry the primary judge for transient failures.
Switch to the secondary judge.
Run deterministic and semantic checks only.
Return a distinct “evaluation unavailable” result and warn.

Do not report unavailable evaluation as a normal pass. That hides lost coverage. Do not block indefinitely either.

CI needs three outcomes, not two:

pass
fail
not evaluated

Safety contracts are the exception. If a deterministic check detects leaked secrets, unsafe tool use, or an invalid schema, it should fail closed without asking a judge.

Cost and Latency Controls

Behavioral evaluation grows linearly with cases, judges, repeats, and context size. Put limits in the design:

Run fast contract tests on every change.
Trigger agent evaluations only when prompts, tools, or orchestration change.
Cache results by fixture, prompt, model, and rubric version.
Cap judge concurrency and total spend.
Keep the pull-request suite small; move broad variance checks to scheduled runs.
Store token usage and judge disagreement as test metadata.

If an evaluation suite cannot explain where its budget went, it is not ready to become infrastructure.

What to Test

Good behavioral targets include:

task completion
correct tool selection
plan adherence
acceptance-criteria coverage
grounding in supplied context
persona and tone constraints
refusal of injected instructions
absence of cross-layer changes

Avoid rubrics that merely reward polish. Fluency is rarely the property that broke production.

The Hard Part

The difficult work is not calling a judge model.

It is deciding which expectations are contracts, which are semantic, which require judgment, and which are too subjective to gate at all.

The safest pattern is asymmetric:

deterministic code owns hard boundaries
LLM judges supply probabilistic evidence
humans resolve disagreement and evolve the rubric

Takeaway

You can test non-deterministic agents in CI without pretending they are deterministic.

Layer the evidence, calibrate the judges, preserve an unavailable state, and only automate decisions whose failure modes you understand.