The stronger model loses when the clock runs

A new benchmark refuses to wait for the model to think. RealtimeBench runs 180 tasks across six kinds of game, and the game carries on while the agent reasons and calls tools. The authors' argument is that an action which would have been correct a second ago can arrive too late to be worth anything, and that no paused benchmark can show this.

The headline result is a reversal. In paired evaluations GPT-6.1-sol outperforms the faster model, Jev, when the games are paused, and falls behind it once they keep running. Which of the two looks better depends on whether the clock is stopped, which means a leaderboard built on paused tasks says little about which model to deploy into a moving environment.

The paper also proposes a way to close the gap, called Two-Timescale Offline Recursive Self-Improvement. A slow teacher studies failures between attempts and tests its corrections in paused games, then writes reusable instructions for a fast student. During play the student acts alone, with no calls back to the teacher, and validation scores decide which updates are adopted.

Two rounds of this raised Jev's average test score from 13.51 to 24.19 on a nought-to-hundred progress scale, and improved all six game types. The instructions also transferred: average scores rose for three other models without retuning. The authors present this as a direction rather than a finished result, the direction being to organise deliberation, learning and action across different timescales so that a capable agent stays effective in a world that does not pause.