Back to Project
D

Doc2Harness

Project Oct 15, 2026

A benchmark that asks models to build a coding-agent harness from documentation, then tests the frozen product with scripted model responses.

Visit primary link 3 sources attached
Doc2Harness diagram showing OpenCode building an agent harness, followed by mock-model evaluation through four observable channels and positive controls.
Authors' diagram of the build-and-evaluate protocol. It describes the method, not independent validation of the reported scores. Source: Yuyao Ge and collaborators, Doc2Harness
agent-ecosystemcoding-agentsverification

What happened lately

Doc2Harness timeline

Doc2Harness separates building an agent framework from evaluating its behavior. A model uses OpenCode to implement a harness from a development document and an interface skeleton. The frozen harness then runs in a separate container against a scripted mock model.

The Apache-2.0 repository releases 53 scored cases and 41 positive controls. The paper’s leaderboard uses a larger, partly held-out suite of 328 test cases; the public subset alone cannot reproduce those full-suite scores. Evaluation requires Python 3.11 or newer and Docker. Scoring an existing harness needs no model API key, while building a new one requires a configured model gateway.