Ai2's StateBench finds frontier models lose track of changing state as inputs grow longer
The benchmark sorts state-tracking tasks into four difficulty levels grounded in Krohn-Rhodes theory; on the hardest, Gemini, Claude and GPT models break down, while recurrent and hybrid models trained on the tasks generalise further than transformers.
Key facts
- Four levels: R-trivial, aperiodic, soluble, general finite
- Grounded in Krohn-Rhodes theory
- Tested models include Gemini 3.8 Flash, Claude Opus 5.5 and GPT-6
- Recurrent and hybrid models generalise further when trained