Two benchmarks surfaced within days of each other:

• MerchantBench — Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations (paper)
• LongHorizon-Harness — Advancing Long-Horizon Agents for Real-World Tasks (paper)

Both target the same gap, and the timing is not coincidental — it is the week three labs shipped long-horizon coding agents.

The failure modes pass-rate cannot see

Run an agent for four hours and "did not finish" is rarely the interesting failure. These are:

Self-contradiction. A decision made at minute ten is silently reversed at minute ninety. The output is internally inconsistent, and each individual step looked fine.

Goal drift. The agent optimises a subgoal it invented after losing the original objective. The work is competent and irrelevant.

Loops with variation. Not literal repetition, which is easy to detect, but the same failed approach re-attempted with cosmetic differences.

None of these produce an error. All produce output. A pass-rate scores them as partial success.

Why "coherence" is the right word

MerchantBench naming long-term coherence rather than task completion is the substantive choice. Coherence is a property of the trajectory, not the endpoint — did any two decisions conflict, did the stated goal drift, was work repeated.

Those are measurable and almost nobody measures them.

Why e-commerce is a clever substrate

Inventory reconciles or it does not. Orders ship or they do not. Prices are numbers.

The domain supplies ground truth without a judge, which removes the largest source of variance in agent evaluation. Compare "write a good summary", where the grader is a model and the metric is a proxy for a proxy.

The research analogue

A literature review over forty papers has exactly this structure. The model that summarises paper one well may, by paper thirty, characterise a method in a way that contradicts its own earlier summary — and nothing flags it, because each summary is individually plausible.

If you have run a long extraction job over a corpus, you have probably shipped a contradiction you never found.

Why this matters right now

Meta, Qwen and Moonshot all shipped agents claiming long-horizon capability this week. These two benchmarks target precisely the failures such claims need to avoid.

The collision between the claims and the measurements is the story of the next month — and until those numbers exist, "long-horizon" is a product description rather than a result.

Sources: MerchantBench · LongHorizon-Harness, surfaced via @_akhaliq