Microsoft Research published Echoverse, twelve synthetic environments for training computer-use agents. Buried in it is a result that should change how people build simulations.

The finding

Compared on live WebVoyager domains:

• Shallow training environments dropped a model from 80% to 75%
• Deep training environments lifted it to 85%

Shallow simulation is not a weaker version of deep simulation. It is actively harmful — worse than not training on synthetic environments at all. The agent learns the shortcuts the shallow environment permits, and those shortcuts do not exist in the real world.

Why existing benchmarks decay

Echoverse names the problem with live-website benchmarks directly: "pages get redesigned, listings and dates roll forward, and hosts throttle or block automated traffic."

That is drift, and it means a score from six months ago is not comparable to one today — not because models changed, but because the test did.

The scale of the gains

A 9-billion-parameter model trained across all twelve environments went from 36.5% to 67.1% average performance. That puts it within fourteen points of GPT-5.4 (80.7%), and on EchoBank and nested filters the 9B model "matches or beats the far larger frontier model outright."

Tasks run five to twenty actions deep. The datepicker world alone renders one date control as six core widgets across ten contexts, plus ten held-out variants.

The verification choice

Grading is grounded in database state rather than screenshots. That sounds like an implementation detail and is not: screenshot-based verification is brittle, subjective and expensive, and it is why so much agent evaluation needs a human or a judge model in the loop.

If the environment owns the database, correctness is a query.

The limitation the authors state plainly

This is the part I would have expected them to bury, and they did not:

more trajectories on the same worlds keep lifting the in-domain average, though the gains keep shrinking, while transfer to the live web flattens outright

In other words: scaling one environment yields diminishing in-domain returns and no improvement in generalisation. Improvements on public benchmarks are incremental, which they attribute to domain coverage rather than architecture.

That is a genuinely useful negative result. It says the path forward is more distinct environments, not more data in existing ones — and it is the sort of finding that usually goes unpublished.

What is released

Four environments — EchoStay, EchoForge, datepicker and nested-filter — with code, data, graded evaluations and the database-grounded verifier, on GitHub and Hugging Face.

The takeaway

If you are building synthetic environments to train or evaluate agents, depth is not a quality nicety. A shallow environment measured worse than nothing, and that result is cheap to reproduce before you invest in building twelve of your own.

Why a benchmark stops measuring

Nothing here requires bad faith. It is the natural consequence of publishing a fixed test into a world that trains on published text.

Source: Microsoft Research — Echoverse