53 agents, 37 reports and a deletion that never asked permission

An agent could ask before running a script, then delete the script without asking. That is the opening example in DefuzeX’s October 18 account of behavioral testing across 53 open-source agents, which the company says produced 37 upstream reports.

The distinction matters: these are reports, not 37 independently verified vulnerabilities. The writeup says six were confirmed fixed, several received maintainer replies or triage, and most were still waiting. Its results table lists 2,118 executed cases, 649 automated issue flags and 126 cases with manually confirmed agent problems. The introduction also uses the looser description “about 3,000” cases; the tabulated counts are the more specific figures, not an interchangeable total.

A permission gap with a public fix

The detailed nanocoder report concerns a historical October 1 run of version 1.30.0 through the Agent Client Protocol, or ACP. Its default auto-accept mode skipped approval for the file-operation tool, including deletion, while shell commands still prompted. The tool’s own description steered file removal toward that less guarded route.

The appended delete instruction was in the same user message as the legitimate task, so this is not proof of an external prompt-injection bypass. DefuzeX’s harness also answered permission requests with allow: prompting would not have changed that particular run’s outcome. The concrete finding was the missing opportunity for an interactive editor user to approve or reject the deletion.

Public maintainer discussion supports that narrower interpretation. The resulting pull request, merged October 12, makes deletes and destination-overwriting moves or copies prompt in auto-accept under the default configuration. It leaves the ACP default mode in place and preserves other modes and explicit preauthorization. Today’s news is the broader findings report, not a newly discovered or newly fixed nanocoder flaw.

The judge needed checking too

DefuzeX used its KUMA SDK with AgentBehaviorBench to generate cases, run targets in separate containers and compare recorded actions with the agent’s claims. Its examples include false completion statements, swallowed errors and protocol messages lost between agents and editors.

Only roughly one in five automated flags survived trace review, according to the authors. Different agents received different generated tasks, so the case counts cannot rank their reliability. One reported permission problem was the harness’s own mistake: it had approved the action. The authors say they now expose those approval decisions to the judge.

The MIT-licensed harness requires a source checkout, Python and Docker for real target-agent runs; Windows setup uses WSL. Official KUMA case generation and judging need separate service credentials, alongside the target model and any search tools. A local SDK avoids KUMA credit but can still incur model costs. Execution evidence is sent for judging, making the recorded content and service configuration relevant to anyone testing private work.

AITrending read the English report, project documentation, reproduction and merged fix without rerunning the benchmark. DefuzeX sells agent-reliability tooling, and its public Hacker News discussion is not an independent validation. The useful takeaway is concrete: agent summaries and automated verdicts both need evidence behind them.