AgentpitBench puts coding agents' paper-money bets and transcripts on Hugging Face

A prediction-market leaderboard is more useful when readers can inspect how the agents reached their bets. AgentpitBench’s new Hugging Face dataset opens that path: its October snapshot describes 51 resolved rounds and 204 agent entries, alongside rationales, timing, reported model identities and redacted session transcripts.

The dataset was created on October 19 at 16:42:20 UTC. The benchmark itself is older: its rulebook records changes beginning October 9 and earlier exhibition activity. Today’s development is the downloadable publication of October records, not a newly started contest.

What the release makes available

The English card lists Claude Code, Codex CLI, Gemini/agy and Grok Build as the four coding-agent routes. Their assigned model and reasoning settings are part of the experiment, and a model_reported field records what each run reports. These are complete agent setups with tools and time limits, not isolated language-model answers.

The operator says each participant receives the same prompt at the same minute and has five minutes to make one bet. The currency is paper-money apUSD, with no real money at stake. The download separates agent entries in bets.parquet, market outcomes and resolution information in rounds.parquet, and redacted logs in transcripts.jsonl. The card provides Python datasets.load_dataset examples and labels the data CC BY 4.0.

Methodology v2.2 describes vendor CLIs running inside bubblewrap sandboxes with fresh home directories, restricted environment variables and access to web research. The benchmark uses the operator’s subscriptions. Downloading its published records is separate from provisioning those CLIs and model accounts to run a similar experiment.

The rules affect what a ranking means

The current headline metric is total paper profit, followed by win rate; a missed bet counts as a loss. The site explicitly says the betting table is not tested for statistical significance. A separate daily probability-forecast sweep remains research data and is not ranked.

The rulebook also records an October 13 switch from forecast accuracy to betting records. It says Claude led the forecast table at that point, while Gemini led the betting table. Earlier entries preserve their original model configurations when later runs change models. Market-selection settings can change within published bounds, with a versioned explanation applying to future rounds. Those details make a season-wide position unsuitable as a simple, controlled claim that one model is always the best forecaster.

There is a commercial relationship to keep visible. SKALE Labs operates both AgentpitBench and agentpit, the exchange used by the agents. Its disclosure says markets mirror Polymarket, that no lab pays for placement and that operator tooling uses Claude. These are the operator’s disclosures, not an independent audit.

The practical addition is access to decisions and evidence: researchers can inspect missed bets, rationales and transcripts instead of relying only on a score. The release does not establish statistically reliable superiority, transferable trading profits or the quality of an agent outside this specific paper-money setting.