All stories
CUGA Ecosystem Community reports

Same model, 30% to 81% task completion — IBM's CUGA team makes the case for choosing your harness

IBM's team reports 73 of 90 AppWorld tasks passed with CUGA, versus 42 with Hermes and 27 with Deep Agents, using the same GLM model but different configurations.

By AITrending3 min read
Sources (3)

Why now A November 3 IBM Research post publishes a same-model comparison and invites improved competing configurations for a rerun.
CUGA author-supplied manager interface shows CRM and Workspace connections, three policies and a draft Revenue Operations configuration.
Project-supplied manager screenshot with illustrative Revenue Operations configuration data and example internal endpoints. It does not show an AppWorld benchmark run or an independently verified business outcome. Source: IBM / CUGA contributors

Changing the agent harness can change the result even when the model stays the same. In a November 3 post, IBM’s CUGA team reports that GLM-5.3-flash completed 81.1% of a 90-task AppWorld sample with CUGA, compared with 46.7% using Hermes and 30% using Deep Agents.

Those are the CUGA team’s own results, not an independent ranking. The team also discloses a crucial qualification: the configurations were different, and CUGA used an AppWorld-specific setup the other harnesses did not.

What the comparison actually measured

The experiment used the Harbor evaluation framework, the same model and the same 90 tasks across 30 scenarios, with one attempt per task. AppWorld simulates work across applications such as email, payments and file storage. The authors describe it as a proxy for some business-automation capabilities, rather than a complete enterprise test.

HarnessTasks passedScenarios with all three variants passed
CUGA73/90, or 81.1%20/30, or 66.7%
Hermes42/90, or 46.7%8/30, or 26.7%
Deep Agents27/90, or 30%2/30, or 6.7%

Results reported by IBM’s CUGA team; configurations were not identical.

CUGA’s setup included task decomposition, focused tool access, Python-based API actions, verification before execution and automatic reflection. Hermes used Harbor’s default setup with AppWorld tools over MCP. Deep Agents used MCP plus middleware for tool selection, summarization, to-do lists, retries and call limits; the team also added two fixes for malformed GLM replies.

The practical recommendation is to test the model and harness together on the work users actually need. These numbers support a specific comparison of configured systems on this sample. They do not establish that CUGA universally outperforms personal assistants, coding agents or research frameworks.

Business actions, with separate policy controls

CUGA is IBM’s Apache-2.0 generalist-agent harness for workflows across applications, APIs and documents. Its English README documents Python 3.12 and uv installation, model-provider and credential configuration in .env, and a local management interface where users connect tools and test a draft before publishing a configuration. Model inference and hosting costs depend on the chosen endpoint.

Its policy system can require human approval for selected tools or apply configured ToolGuards before execution. Connected services and their credentials determine what the agent can reach; an applicable, enabled guard is required for a runtime check. A natural-language instruction alone does not create that check.

The benchmark did not use policies. The separate policy-playground demonstration in IBM’s post therefore should not be treated as a measured explanation for the AppWorld scores. Likewise, the project’s illustrated management interface uses sample configuration data rather than proving a successful live business run.

The authors link a Harbor fork as their reproduction starting point and invite stronger Hermes or Deep Agents configurations, promising to rerun the same tasks and publish the outcome whichever harness wins. For teams choosing an agent stack, that invitation is as useful as the headline score: the open question is which configuration works best for their workload.

Related reading