Smaller tool outputs, bigger bills? New agent benchmark tests the whole session

The compression ratio can hide the cost

A new community benchmark asks whether shrinking an AI agent’s tool output actually makes the whole session cheaper. In one published comparison, it did the opposite. Beyond Compression Ratios, uploaded to Hugging Face on October 17, measures total session prompt tokens alongside answer correctness rather than treating a smaller individual response as the final result.

On 108 test questions using real MCP git, filesystem and SQLite servers, the author’s qwen3:4b-instruct runs recorded 74 correct answers with raw output versus 73 with the lp_safe compression condition. Total prompt tokens rose from 1,241,758 to 1,453,407, an increase of about 17%. Those counts match AITrending’s aggregation of the uploaded episode records; this checks the recorded arithmetic, not the correctness labels or an independent rerun of the agents.

The author says the proxy could greatly shrink individual outputs, but agents fetched omitted information back or lost the information they needed. The practical hook is that an impressive per-response reduction can coexist with extra model calls, retrieval work and higher session token use. The published numbers measure tokens, not a monetary invoice: any actual bill depends on the provider and pricing.

Logs to inspect, with limits to keep visible

The English dataset card describes 1,832 logged episodes across real and synthetic task sets, mostly using Qwen3 4B Instruct through Ollama 0.35.0. Real-server questions use the Flask 3.1.1 repository and Chinook sample database, with gold answers computed by code. Records include model calls, shown tool text, final answers, token counts and author-supplied correctness labels. The CC BY 4.0 data release also includes task specifications and a manifest explaining the proxy versions used.

Other conditions cover learned compression, Headroom, truncation and the effect of merely listing a retrieval tool. They are experimental conditions, not interchangeable implementations or a general ranking. The separate output-level measurement file estimates tokens as characters divided by four; the session metric is described as summed Ollama prompt-evaluation counts. The two should not be treated as the same measurement.

The limitations are material. These are small-model, single-question sessions with only a few tool calls, so larger models and long workflows may behave differently. A documented proxy bug dropped server environment variables and caused all 16 memory-server tasks to fail in one synthetic learned-compression condition; the author advises filtering those tasks for a comparison without that bug. Synthetic tasks also need their world generator, while the real-server tasks can be rerun with public servers and data.

For builders considering context compression, the release offers inspectable traces and a concrete evaluation target: compare total session tokens and successful answers. It does not establish that compression always fails, that every retrieval mechanism adds cost, or that the author’s labels have been independently validated.