MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations targets a failure mode most agent evaluations do not touch: consistency across a long sequence of related decisions.

Correctness versus coherence

Most agent benchmarks test a task in isolation. Give the agent a goal, let it act, check the outcome. That measures correctness.

Running a business is a different shape. You set a price on Monday and it constrains what is sensible on Friday. You promise a customer a delivery window and that commitment persists. You choose a supplier and switching has a cost you incurred by choosing.

An agent can be individually correct at every decision and still produce an incoherent operation — repricing daily in ways that contradict last week, making commitments it will not honour, oscillating between strategies because each day's context looks slightly different.

Why e-commerce is a reasonable testbed

It has the properties you want in a coherence benchmark:

• Decisions are genuinely coupled. Pricing, inventory, and promises constrain each other in ways that are hard to fake.
• Outcomes are measurable in units nobody argues about. Margin, stockouts, fulfilment rate.
• The horizon is long enough for drift to appear but short enough to simulate.

That last property is the practical one. You cannot run a benchmark that takes a real year.

The measurement difficulty

Coherence is harder to score than correctness, because the standard is internal. There is no key that says the Friday decision should have been X — only that it should have been consistent with Monday's, given what changed in between.

That means the benchmark has to encode what counts as a legitimate change of course. An agent that never revises is not coherent, it is rigid. An agent that revises whenever the wind shifts is not adaptive, it is incoherent. The interesting behaviour is in between, and specifying "in between" is the hard part of building this kind of evaluation.

Why it matters beyond retail

Every deployed agent that runs longer than a session has this problem. Customer support agents that contradict yesterday's answer. Coding agents that refactor toward a pattern and then away from it. Research assistants that recommend a direction and then quietly abandon it.

These do not show up in single-task evaluation, and they are among the most common reasons pilots do not become deployments. Benchmarks that surface them are more useful than another point of accuracy on a task that already saturates.

The paper is available on Hugging Face.