All stories
Doc2Harness Launches Community reports

Doc2Harness asks AI to build the agent itself; the best reported score is 53%

The new benchmark tests model-built coding-agent frameworks, with a public subset that exposes practical failure modes but cannot reproduce its full leaderboard alone.

By AITrending2 min read
Sources (4)

Why now The English paper was submitted October 14 at 15:24 UTC. The project's available code documents a partial release of 53 scored cases and 41 positive controls, giving agent builders a concrete evaluation route.
Doc2Harness diagram showing OpenCode building an agent harness, followed by mock-model evaluation through four observable channels and positive controls.
Authors' diagram of the build-and-evaluate protocol. It describes the method, not independent validation of the reported scores. Source: Yuyao Ge and collaborators, Doc2Harness

Build the framework, then test it

Doc2Harness turns a familiar coding-agent benchmark inside out. Instead of asking a model to fix a repository while an existing agent handles the machinery, it asks the model to build that machinery: the harness that manages tools, model requests, sessions and permissions.

The authors report that the strongest of nineteen tested models averaged 53.05% on their full 328-case suite, with three independently built harnesses per model. That is a result for this specific harness-engineering protocol, not a claim that the model fails half of all coding tasks. AITrending has not independently reproduced the scores.

The paper was submitted on October 14. Its public repository is explicitly a partial release: 94 scenarios comprising 53 scored cases and 41 positive controls. The remaining cases are held out, so downloading the public subset does not let users reproduce the full-suite leaderboard alone.

A mock model makes the harness observable

In the build stage, OpenCode 1.17.0 gives the model a development document and an interface-only skeleton of about seventy lines. The protocol allows up to two hours, 24 turns and three submissions. After a form check accepts a product, it is frozen rather than iteratively repaired using the scored tests.

Evaluation moves that harness to a separate container and drives it with scripted mock-model responses. Tests inspect requests to the model, files on disk, process exit and command output. Positive controls distinguish a real boundary from a broken tool: blocking a forbidden file operation is not meaningful if permitted operations cannot work either.

Failures can also be correlated. A single implementation defect may break several related cases, so the percentages should not be read as counts of independent engineering mistakes.

A usable subset for agent builders

The Apache-2.0 repository requires Python 3.11 or newer and Docker, with standard-library Python code. Users can build the evaluation image and point the catalog runner at an existing harness without a model API key because scoring uses the mock gateway.

Building a new harness is a separate route through OpenCode and a local LiteLLM gateway holding the user’s provider key; those model calls may incur charges. For teams building coding agents, the immediate value is a concrete set of observable behavior checks, alongside a leaderboard whose full reproduction remains limited by the held-out suite.

Related reading