Mirasim
Get Mirasim
← Blog

Field notes3 min read

Let simulated users test what the agent built: get evidence, not a model's verdict

An agent that reviews its own work only grades the code diff. Knowing if a person can finish the task takes somebody actually using what it built.

A Mirasim session where a reviewer leaves notes on what the agent producedA Mirasim session where a reviewer leaves notes on what the agent produced

Why an agent's self-review is not enough

Agent reviews its own code
Someone uses what the agent built
What the review checks
Whether the code compiles, uses consistent names, and handles edge cases
Whether a person can actually complete the task using the product
How the check is done
The agent reads and inspects the source code changes
Running and operating the real build that the agent just produced
What question it can answer
Whether the code change itself is written correctly
Which step stops users, which line is unclear, or which button nobody finds
This comparison shows that "The diff is correct" and "the feature is usable" are different claims, and only the first can be checked by reading code.

Ask an agent whether its own code change is good, and you will get a competent review of the code diff — confirming that the code compiles, naming is consistent, and edge cases are handled — all of which may be true without answering whether the feature actually works for a person. The questions that matter early on are much more basic than code review: which step nobody gets through, which sentence nobody understands, and which button nobody finds. Those usability failures surface as soon as somebody actually uses the build, in minutes rather than weeks, before your effort is sunk.

How a simulated-user evaluation runs

  1. You invoke $eval to start testing

    Evaluation is an on-demand skill you trigger, not an always-on review

  2. Simulated users try the real build

    Simulations run locally on the real build the agent just produced

  3. Evaluation returns usage evidence

    Screenshots, traces, commands, and reports land in the artifacts panel

  4. Evidence stays anchored to its run

    Tied to the host machine and run, and previewable right beside the session

This flow shows how running $eval returns traceable evidence rather than a score, because any conclusion that cannot be traced back to a run is treated as no conclusion.

How evaluation evidence feeds into the next steps

  1. Round-one evidence is saved

    The evidence stays stored with the project it belongs to

  2. Evidence becomes the next tasks

    Failed steps become agent tasks without anyone retyping tickets

  3. The agent fixes the exposed issues
  4. Run round-two evaluation

    Compare round two against round one instead of overwriting it

This loop shows how evaluation drives fixes without anyone retyping findings as tickets, while keeping round-one evidence so round two can answer "did the fix work".

Simulated users are not a substitute for real users, and this evaluation does not claim to predict what your customers will do. Instead, simulated users offer a fast, repeatable way to catch failures that are obvious in hindsight — the flaws you would be embarrassed to have shipped, and would otherwise have shipped.

Keep reading