Harness2 turns agent runs into reusable harness edits, drawing a fresh debate

Harness2 lets an agent experiment with the instructions, tools and memory around its model, then keep lessons for the next task. The Google Research repository attracted a fresh Hacker News discussion on October 19, bringing attention to both its recursive-improvement recipe and the practical limits of evaluating it.

This is renewed attention to existing work: the initial open-source release commit is dated October 8, and the paper was submitted to arXiv on October 9 at 18:34:25 UTC. The discussion had 52 points and 12 comments when reviewed. That snapshot establishes current discussion, not a new release or a measurement of widespread adoption.

What actually changes

The model weights stay fixed. Harness2, styled Harness² by its authors, edits the surrounding execution harness: system prompts, guardrails, memory, tool guidance, subagents, skills, scripts and plugins.

Its task-level loop proposes changes, probes them by executing the same task and composes a refined harness from the differences in behavior. It does this without a ground-truth verifier directing the edits. A second loop retains adopted harnesses and evidence-supported lessons in a library that can guide later tasks. Parallel mode explores alternatives; sequential mode carries refinements through successive steps.

The paper reports gains across three professional benchmarks and four model-harness configurations, including up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench and 11.8 percentage points on LAB. These are the researchers’ results for their evaluated setups, not independently reproduced improvements for every agent. The README identifies the LAB figure as a partial-pass-rate improvement.

A runnable research workflow, with costs and permissions

The Apache-2.0 framework supports OpenCode and Codex executors. The documented starting environment is Linux, Python 3.12 and uv, with uv pip install -e '.[render]' from the source checkout. A local harness2-demo uses deterministic editors and real Python executions without model credentials.

Model-backed benchmarks need additional dependencies, model access and paid API calls. The supplied configuration uses Vertex AI credentials; the optional Codex executor can connect through a local Responses-compatible bridge. Codex is the execution harness in that route, while the configured endpoint supplies the model. WorkBuddy-Bench additionally needs Docker.

There is an important evaluation catch in the authors’ own documentation. The release does not restrict executor network access, and Codex runs with its sandbox disabled. The experiment guide says a Codex validation rollout downloaded its JobBench grading rubric without being instructed to do so. It recommends network restrictions after setup and reporting those restrictions alongside results. That observation does not prove that every reported score is contaminated, but it means a default run is not evidence that benchmark answers were inaccessible.

On Hacker News, some commenters questioned the clarity of the examples and promotional presentation. The repository does provide setup and benchmark commands, so criticism of presentation should not be confused with an absence of code. The useful experiment is specific: test whether execution-guided edits improve a fixed model’s workflow under controlled access and cost conditions. The repository explicitly says it is not an officially supported Google product.