TCSExam, a 100-task theoretical computer science benchmark, puts GPT-6 Astra first at 71.9
The arXiv paper, with Andrew Chi-Chih Yao among the authors, checks answers with the Lean kernel or an executable checker; Opus 5.5 leads the 20-task Lite set at 72.3, and the two sets rank eight shared agents almost identically.
Why it matters
TCSExam tries to measure whether language-model agents can contribute to theoretical computer science, not just solve exam-style problems. It spans theorem proving, solver engineering and constructions, plus direct attempts on open conjectures, with answers checked by the Lean kernel or an executable checker. GPT-6 Astra scores 71.9 on the 100-task Full set, Opus 5.5 leads the 20-task Lite set at 72.3, and the sets rank eight shared agents almost identically (Spearman 0.93). The authors also report improved solver outputs and mathematical constructions from the evaluations. Watch whether agents begin settling open conjectures.
Key facts
- Submitted to arXiv on 5 Nov 2026 (2611.07962)
- TCSExam-Full has 100 expert-reviewed tasks; TCSExam-Lite has 20
- Submissions are checked by the Lean kernel or an executable task checker
- Eight agents evaluated on Full, nine on Lite
- GPT-6 Astra leads Full at 71.9; Opus 5.5 leads Lite at 72.3
- Full and Lite rankings agree across eight shared agents (Spearman 0.93)