How do you run AI agent tests in CI so a bad change fails the pull request?

Pick the cases from real traffic

A suite invented at a desk tests the paths you already thought about. Take the cases from recorded sessions instead: the twenty flows that make up most of your traffic, plus every session that caused an incident. The incident ones are the highest-value tests you will ever have, because each is a regression somebody has already paid for once.

Keep the suite small enough that people wait for it. Under ten minutes is a gate; over thirty is a thing engineers learn to merge around. Split it if you need to: a fast subset on every pull request, the full set nightly.

Make the run deterministic enough to compare

A gate only works if a red result means the change was bad, not that the dice came up differently. Pin the model version explicitly rather than taking a floating alias, set temperature to 0 where the task allows it, and seed the environment from a fixed seed so every run starts from the same world.

Some variance survives all of that, and pretending otherwise produces a flaky gate. Handle it by asserting on things that are stable (which tool was called, with what arguments, what state resulted) rather than on the exact sentence, and by running the handful of genuinely stochastic cases more than once and requiring a pass rate instead of a pass.

Run against a fake backend, not a shared one

CI is the worst possible consumer of a shared staging system: it runs on every push, in parallel, from branches, at 3am. Two pull requests hitting the same staging database at once produce failures that belong to neither of them, and the team learns to re-run the job until it is green, which is the same as not having a gate.

Give each job its own environment instead: containers started per run, seeded from fixtures, thrown away at the end. That also means the CI job needs no production credentials at all, which removes a standing secret from a system that runs untrusted branch code.

Compare against a baseline, not an absolute

Absolute thresholds age badly. "Score must exceed 0.8" is either permanently red or quietly meaningless within a quarter. Run the candidate (the branch) and the baseline (the current main) over the same cases and report the difference: which cases changed verdict, in which direction, and what the agent did differently.

Present that diff per case. "Overall quality fell 3%" is not actionable; "case 14 stopped calling check_refund_eligibility and issued the refund anyway" is a bug report.

Publish a status check, then make it required

The mechanics are ordinary CI: run the suite on pull_request, and report a commit status or check run with a name and a link to the results. Any CI system can do this, and so can a webhook from a platform that runs the suite for you.

The step teams miss is the last one. A status check is advisory by default. GitHub only holds a merge on checks that the repository names as required, under branch rules, and no external tool can set that for you (it should not be able to). Until you add it there, a red gate is a red mark that anyone can merge past.

Know what required means before you turn it on: a required check gates every write to the branch, not just merges. If the gate only runs on pull requests, a direct push to the protected branch has no check to satisfy and GitHub rejects it outright. That is correct for a team already working through pull requests, and a locked door for anyone else.

Decide what red means, in advance

Agent suites produce judgement calls more often than unit tests do, so write down who reviews a red gate and what they are allowed to do about it. A gate nobody can override gets deleted; a gate anybody can override with no record is decoration. The workable middle is an explicit approval, by a named person, recorded on the pull request.

Budget the cost too. Every gated run spends model tokens and machine time on every push, so a suite that runs on every commit of a busy repository is a line item. Running on pull requests rather than pushes, and on the fast subset by default, is usually the difference between a bill people accept and one they turn off.

Where Mirrors fits

Mirrors provides the per-run environment and the candidate-versus-baseline comparison. Its CI gate connects to a repository, and on each pull request it fetches the agent code at a configured path, runs a bound eval set against the environment with the candidate agent, and diffs the results against the baseline. The result lands as a status check with a per-case diff behind it, which you approve or reject.

It deliberately reports rather than blocks, and it will not edit your branch rules. If you want a regression to hold the merge, you add the check as required yourself, which is the same last step described above and the same one that catches people out.

Choosing the cases, deciding what red means, and writing the assertions that encode your business rules stay with you. Those are the parts that make the gate worth having.

What this does not solve

A gate is only as good as its cases. It proves that the change did not break the twenty flows you pinned; it says nothing about the twenty-first, and agents fail in the long tail more than in the middle. Treat coverage as something to grow every time production surprises you.

Model nondeterminism does not fully go away. Temperature 0 is not a guarantee, providers change model behavior under a stable alias, and a suite that is 2% flaky across fifty cases fails roughly two-thirds of the time on noise alone. That is the arithmetic that turns a gate into something people bypass, and it argues for fewer, stabler assertions rather than more sensitive ones.

And the gate cannot tell you whether the change was a good idea. It tells you what changed. Somebody still has to decide whether the new behavior is better, which is why an approval step is a feature rather than a missing automation.