How do you test an agent without using real customer data?

Why the masked production copy is the wrong default

The instinct is to snapshot production and scrub it. It is worth naming what that actually buys, because it is less than it looks. The copy is still derived from personal data, so it stays in scope for whatever regime you are under; masking is famously incomplete on free text, and an agent test set is mostly free text; and quasi-identifiers survive scrubbing, so a small enough dataset can be re-identified from fields nobody thought of as identifying. Every refresh repeats the whole exercise.

Generating from the shape avoids all of that, and it is usually less work after the first week. The one thing it costs you is the long tail of genuinely weird real records, which is why the last step below exists.

Keep the shape, invent the values

Fidelity for an agent test lives in structure, not in values. What has to be right: the entities and how they relate, referential integrity (every order belongs to a customer who exists), realistic cardinality (most customers have one order, a few have four hundred), field formats the agent parses, the distribution of statuses, and the presence of the awkward cases.

What does not have to be right: the actual names, emails, card numbers, addresses, or free-text notes. An agent that behaves correctly for "Dana Whitfield, order 1182" behaves the same for the real customer, because nothing in its logic depends on which human it was.

Put the awkward cases in on purpose, because they are what a random generator misses: the empty result, the single-item result, the pathologically large result, the record with a null in the field everyone assumes is set, the unicode name, the customer whose order was already refunded.

Where the shape comes from

Three sources, in descending order of how much you already have. A database schema or an OpenAPI spec gives you structure directly and involves no personal data at all. Recorded traces give you the most accurate picture of what the agent actually sees, including the fields the spec forgot, but only if they are redacted at the collector, before they leave your network, so the recording that reaches the tool never contained the value in the first place. And where neither exists, a domain expert writing down the rules by hand for an afternoon beats a generator guessing.

The point about redaction is the one that matters legally: redacting after storage means you stored it. Structure survives redaction perfectly well, so a trace where every value has been replaced by a typed placeholder is still enough to learn the shape from.

Seed deterministically

Generate from a fixed seed so the same seed produces the same world, and record the seed with the test run. Without that, a failing test cannot be reproduced, a comparison between two agent versions is confounded by the data moving underneath them, and "it passed on my machine" becomes literally true and completely useless.

Reset between runs as well. A suite where run 3 inherits the refunds issued by run 2 will produce failures that belong to the previous test.

Keep a small hand-written set

Generated data is good at the middle of the distribution and bad at the tail, and agents fail in the tail. Keep a handful of cases written by a person who knows the domain: the ones that caused incidents, the ones support keeps escalating, the ones where the correct behavior is counter-intuitive.

When production does surprise you, add the case, invented values and all. That is how the suite grows in the direction reality is pushing it, and it costs one commit per surprise.

Do not forget the traces themselves

Test data is only half of it. The recordings you use to build the suite are also customer data, and they are often the copy nobody has a policy for. Redact at the collector, decide a retention window and enforce it, and check what any platform you send traces to does with them: whether they are used for training, how long they are kept, and whether model calls made on your behalf are routed with zero data retention.

It is a shorter conversation with a security team than a masked database copy, and it is the conversation that actually determines whether you are allowed to do any of this.

Where Mirrors fits

Mirrors learns the shape and fabricates the values, which is the first three steps above. It builds an environment from your traces, your code, your OpenAPI specs, or your docs, and seeds it with invented records that keep the entities, the relationships, and the field shapes of the real thing. Redacted traces are enough: the structure is what it reads, not the values, so a trace whose values were replaced at the collector still produces a usable environment.

Seeding is deterministic by design. The same seed and the same instructions produce a byte-identical world, which is what makes a replay reproducible and what lets two versions of an agent be compared without the data shifting underneath them.

The hand-written tail cases are still yours to write, and so is the retention policy for your traces. No environment builder can tell you how long you are allowed to keep a recording.

What this does not solve

Invented data will not contain the real record that breaks you. The customer whose address is four lines of emoji, the order with a negative quantity from a 2019 migration, the duplicate that should not exist: a generator will not invent those, and the agent's handling of them stays untested until one shows up in production and you add it by hand.

Learning the shape from traces inherits every gap in the traces. A field that only appears in a code path nobody exercised is a field the environment does not know about, and a tool with no recordings behind it is rebuilt from its declaration alone, which is a weaker basis and should be treated as one.

And none of this is a compliance opinion. Fabricated data removes the personal data from the test environment; it does not decide for you whether the traces you collected are lawful to hold, or for how long. That is a conversation with your own counsel, not with a testing tool.