Orchard: a 3B agent hitting 69.7% on SWE-bench Verified

Engineering

Orchard: a 3B agent hitting 69.7% on SWE-bench Verified

A three-billion-parameter agent matching systems ten times its size is interesting. That Microsoft released the training recipes is the more useful part.

An empty open-plan office at night seen through glass, rows of dark monitors

Published

July 27, 2026

Reading time

3 minutes

Perspective

Engineering

Topics

agents · microsoft · evaluation

Microsoft Research has released Orchard, an open framework for building agentic systems. The headline number is unusual enough to lead with: 69.7% on SWE-bench Verified — 73.0% with value-model reranking — using roughly 3 billion active parameters.

Microsoft describes this as approaching systems "10 times larger."

The problem it names

The framing in the announcement is worth quoting, because it is a rare admission of a structural issue:

state-of-the-art agentic systems often require proprietary infrastructure, including custom sandboxes, closed training pipelines, and proprietary datasets that most researchers and practitioners cannot access or reproduce.

That is the actual barrier in agent research. Not model capability — infrastructure. If every lab builds its own sandbox and harness, no two published numbers are comparable, and none of them are reproducible by a third party.

What is in it

The core is Orchard Env, a lightweight Kubernetes environment providing isolated, reusable components for running agents at scale. On top of it sit three domain implementations:

  • Orchard-SWE — software engineering; the 69.7% figure above
  • Orchard-GUI — web navigation; 68.4% average across WebVoyager, Online-Mind2Web and DeepShop
  • Orchard-Claw — productivity assistance; 59.6% of tasks within three attempts, rising to 73.9% paired with the stronger ZeroClaw agent system

Training data, evaluation methods, code and three training recipes are on Hugging Face and GitHub.

The design decision that matters

Orchard trains agents "directly in the harness that it will be deployed with — OpenClaw, Codex, ZeroClaw, or others."

That is a departure from the usual arrangement, where a model is trained in one setting and deployed into another, and the gap between them is absorbed by prompt engineering. Training in the deployment harness removes that gap, and it explains how a 3B active-parameter model competes with much larger ones: it is not a better model, it is a better-fitted one.

What the parameter count does and does not mean

"About 3 billion active parameters" describes activation, not total size — a mixture-of-experts model activates a fraction of its weights per token. So this is not a claim that a 3B dense model matches a 30B dense model.

It is a claim about inference cost, and on that axis it is a strong result. Serving cost tracks active parameters, not total.

The caveat the post does not state

There are no stated limitations in the announcement, which is itself worth noting. SWE-bench Verified is a curated subset, and a system trained in a specific harness and evaluated in that harness is measuring something narrower than general capability.

The honest read: this is strong evidence that harness-matched training is efficient, and weaker evidence about how these agents behave in a repository nobody tuned for.

Why the release matters more than the number

Benchmark leads change monthly. An open framework with published recipes changes what the rest of the field can do — and the reproducibility problem Microsoft names in its own opening paragraph is the thing actually holding agent research back.

Source: Microsoft Research — Orchard

Continue reading

More from COREXA