Back to Project
S

StateBench

Project Nov 3, 2026

Ai2's four-tier benchmark and reproducible generator for tracking changing entity and variable state across long contexts.

Visit primary link 3 sources attached
Ai2 illustration of colored paths changing position across a sequence, accompanying its StateBench state-tracking announcement.
Ai2-supplied StateBench illustration. The colored paths illustrate changing state; this is not a plot of measured model accuracy. Source: Ai2 / StateBench authors

What happened lately

StateBench timeline

StateBench distinguishes immutable assignments, latest-value tracking, self-referential updates and general finite state tracking with cross-variable dependencies. It renders linked assignment traces into different formats to compare models on the same underlying state changes.

The authors evaluate pretrained and frontier models alongside small models trained directly on the tasks. Reported failures concern these controlled evaluations, not every possible use of a model. The framework currently assumes a fixed number of variables. Its paper was submitted on November 2, 2026, and Ai2 published an explanatory announcement on November 3.