Agent regression test
An agent regression test replays a recorded agent session against a changed prompt, tool, or model and checks that the behavior a team already relies on still holds. It asks whether a known-good interaction still works after a change, rather than scoring how good the agent’s answers are in general.
Last updated 2 August 2026
How one works
The shape is always the same. Record a real session, including every tool call the agent made and what each one returned. Rebuild the world that session ran in, so the same lookups return the same records. Replay the session against the version you are about to ship. Then compare the two runs and decide whether the difference is the one you intended.
The comparison is the hard part, and it is where most first attempts fall over.
Why output diffing fails on agents
A regression test for a pure function compares a returned value to a stored one. An agent has no such value. Ask the same model the same question twice and you get two different sentences that mean the same thing, and both are correct. Sampling temperature, retrieval order, tool latency, the current date, and a silent provider-side model update all move the text without moving the behavior.
So a string diff fails constantly while the agent is fine. Teams respond the way anyone responds to an alarm that is usually wrong: they stop reading it, then they delete it. A test that cries wolf is worse than no test, because it also spends the budget a real test needed.
Assert on behavior, not on prose
What is stable across runs is what the agent did, not how it narrated it. That is what an agent regression test should hold to:
- Which tools it called, with which arguments, in what order.
- The state the systems ended in: the refund exists, the ticket is closed, exactly one email went out.
- Invariants that hold regardless of phrasing: no refund above the approval threshold, no write to a record the session never mentioned.
- Bounded properties of the answer, checked by a rule or a grader model: it names the order id, and it does not promise a delivery date.
Notice that the first item is a diff over tool calls rather than over text. That diff has its own name and its own failure modes.
How it differs from an eval
The two words get used interchangeably and they answer different questions. An eval measures quality across a set of cases and gives you a score you want to move up. A regression test protects one behavior that already works and gives you a pass or a fail. Most teams need both, at different moments: evals while you are trying to make the agent better, regression tests while you are trying not to break it.
| Compared on | Agent regression test | Eval |
|---|---|---|
| Question it answers | Did this change break something that worked? | Is this version better than that one? |
| Input | Recorded real sessions | A curated case set, often written for the purpose |
| Result | Pass or fail | A score, or a distribution of them |
| A new behavior appears | Flagged for a human to accept or reject | Rewarded if it happens to score higher |
| Where it runs | A merge gate in CI | A benchmark you rerun while iterating |
An eval that fails a case it used to pass is doing regression testing, badly: it tells you the average moved, not which behavior broke.
Where the sessions come from
A regression test is only as good as the session it replays, and the sessions worth keeping are the ones that already surprised someone: the bug that paged the on-call, the edge case a customer found, the workflow nobody wants to touch. Production traces are the usual source.
The catch is that replaying a trace needs the systems that trace called, answering the way they answered then. That is the part teams do not have, and why agent regression testing is usually blocked on an environment rather than on a test runner. Mirrors exists for that half: it rebuilds the systems from traces, code, or docs and replays the session against them, so redacted traces and systems with no test instance are still testable.
Related terms
- Agent staging environment: A non-production copy of the systems an agent calls, so its actions land on fabricated data.
- Tool-call drift: The gap between the tool calls in a recording and the calls a replay makes after a change.