Artificial intelligence / news brief
Ai2 releases StateBench for long-context state tracking in language models
- Event
- Published

Ai2 released StateBench, a four-level benchmark with code and data for tracking changing variables across long documents. Ai2 reports that Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6 Sol and Luna fell to 13% accuracy or lower on the longest general-finite tasks, while directly trained recurrent and hybrid models generalized farther.
Why it matters
The benchmark separates retrieval from maintaining evolving state, but its synthetic tasks and author-run scores do not yet establish performance on real documents or agents.
Sources
- StateBench: Testing how language models track state changes over long documents (external site)Ai2 · primary source ·
- Ai2 Releases StateBench to Measure Long-Context State Tracking (external site)Unite.AI · coverage ·