Reading a million tokens is not remembering what changed: Ai2 introduces StateBench
Ai2 reports that four frontier models fall to 13% accuracy or less on its longest, hardest state-tracking problems, while targeted training reveals major architecture differences.
A long context window can hold a sequence of events without reliably keeping track of what those events changed. Ai2’s new StateBench benchmark separates those abilities, testing whether language models can maintain the current values of entities through increasingly complicated updates.
In its November 3 announcement, Ai2 reports that Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna all fell to 13% accuracy or below on the longest problems in the hardest category. These are the researchers’ results on synthetic state-tracking tasks, not a general score for coding, conversation or real-world reasoning.
Finding a fact versus following a chain
StateBench has four levels. The easiest keeps values fixed: find what a variable was originally assigned. The next allows changes but only requires the latest value. A third adds updates that depend on the variable’s previous value, like a running score. The hardest lets one variable copy another variable’s current value, creating chains of dependencies.
The distinction matters for tasks such as reading code or following who holds an object in a story. Retrieving the most recent sentence is not enough when its meaning depends on several earlier changes. The benchmark derives easier versions from the same underlying assignment trace, making comparisons less dependent on unrelated differences between examples.
Training changes the picture
Ai2 also trained 275-million-parameter models directly on these tasks. It reports that recurrent Gated DeltaNet and hybrid models generalized much farther than transformers on several levels. On the hardest level, the more flexible Gated DeltaNet variant stayed near perfect to roughly ten times its maximum training sequence length. The tested transformers did not learn that level even with direct training.
That does not make every failure of a pretrained hybrid an architectural impossibility. The researchers explicitly distinguish what a model can learn under targeted training from what ordinary pretraining has taught it. Formatting also affected results: the same underlying task could be much harder when expressed as spreadsheet edits.
The paper was submitted on November 2; the November 3 article explains the work rather than establishing a later paper submission date. Public code documents reproducible trace generation and multiple surface formats. One important boundary remains: StateBench assumes a fixed set of variables, while real conversations and documents can introduce new people, objects and state over time.