How do you safely test an agent that issues refunds, deletes, or sends?

The four bad answers, and why teams reach for them

Reading the code and thinking hard about it is not a test, and it is what most teams are actually doing. Dry-run mode (a flag that makes the tool log instead of act) tests the agent but stops testing the tool, so the two drift and the bug lands in the half you stopped exercising. A shared staging system works until two people use it at once, or until it is empty and no refundable order exists. Testing in production with a small blast radius is how you find out that "small" was measured before the retry loop.

What they have in common is that the agent still has to reach a real system for the test to mean anything. The way out is to stop needing one.

Separate the decision from the execution

This is the highest-value change on the page and it is free. Have the agent emit a structured proposal ("refund order 1182, $40.00, reason: item damaged") and have a thin, dumb executor perform it. The executor holds the rules that must never be wrong: caps, allowlists, idempotency keys, and a required approval above a threshold.

Now the interesting half is testable in isolation. The agent's decision is a value you can assert on, deterministically, thousands of times, with no side effect anywhere. The executor is ordinary software with ordinary unit tests. Most of the risk was never in the model, it was in the fact that the model's output went straight to an irreversible call.

Give the agent a fake backend, not a fake response

For end-to-end runs, the agent should still call tools, and those tools should still do something. The difference is what they do it to. Point them at a service that implements the same contract over invented data: a real store, real writes, real state, none of it connected to anything that matters.

The state part is what makes destructive flows testable at all. A refund that is issued has to be visible to the next read; a delete has to make the next lookup fail; a second refund on the same order has to be rejected the way the real system would reject it. Fixed canned responses cannot express any of that, which is why a response-matching mock stops being enough exactly at the point where the risk starts.

Assert on the call, not on the prose

The dangerous part of a destructive run is the arguments. Assert that issue_refund was called exactly once, that the amount was 40.00 and not 4000, that it targeted order 1182, and that no cancel_order call happened alongside it. Assert on the resulting state too: one refund row exists, the order is not closed twice, the email queue holds one message and not eleven.

Write the negative cases as deliberately as the positive ones. The agent should refuse a refund above the order total, refuse a second refund on an already-refunded order, and refuse to act on an order id it invented. Those are the cases production will find for you if the suite does not.

Make sends and deletes observable instead of real

Outbound effects need a sink rather than a stub: an SMTP catcher or a webhook recorder that accepts the send and lets you assert on it. Mailpit and similar tools do this in one container, and the assertion becomes "exactly one message, to this address, containing the refund amount" instead of a hope.

For deletes, prefer a backend where the delete really happens and the reset is what undoes it. A soft-delete flag in the test double that the real system does not have is a difference the suite will eventually paper over.

Keep the production guardrails anyway

A green suite is evidence about the agent, not permission to remove the seatbelt. Keep the executor caps, keep idempotency keys so a retry storm cannot issue the same refund twice, keep human approval above a threshold, and keep a kill switch that stops the tool rather than the deploy.

Testing tells you what the agent does with the cases you thought of. Guardrails are for the ones you did not.

Where Mirrors fits

Mirrors is the fake backend in the third step. It rebuilds the systems the agent calls from your traces, your tool code, or your API docs, seeds them with fabricated data, and keeps state through a run so a refund is visible to the next read and a delete makes the next lookup fail. Between runs the world resets to the same seeded starting point, which is what lets you replay the same session a hundred times and compare the results.

That is also what makes a recorded production session usable as a test case. The session that paged you can be replayed against a new prompt or a new model, and the refund it issues happens for real, against nothing real.

The decision-versus-execution split, the assertions, and the production guardrails are yours. They are about your business rules, and they are the part worth doing first whether or not you ever build an environment.

What this does not solve

A green run against a rebuilt environment is not proof that your production guardrails work. The caps, the idempotency keys, and the approval step are code that has to be tested on its own terms, and an environment that behaves correctly will never exercise the case where the real payments API times out after charging the card.

It also cannot produce a failure mode it has never seen. If the real refund API rejects requests over $10,000 with a specific error and that has never happened in your traces or your docs, the copy will not do it either, and the agent's handling of it stays untested.

And a fabricated environment says nothing about permissions in production. An agent that behaves perfectly against a copy can still be holding credentials that let it refund an order it should never have been able to see.