ETH Zurich and Berkeley: RL post-training on 144 tasks improves a robot policy on unseen scenes and objects
In the MolmoSpaces simulator, critic-guided methods (BoN, OGPO) gain up to 58 percentage points on unseen scenes and 39 on unseen objects over the base policy; single-task post-training degraded other tasks. The paper is an ICLR 2027 submission.
Why it matters
Post-training a generalist robot policy on one task usually makes it better there and worse elsewhere. This study varies the number of training tasks from 1 to 144 and finds that critic-guided reinforcement learning shifts from forgetting to generalisation as diversity grows. Scene diversity helps new scenes and object diversity helps new objects, with the strongest transfer when added object categories resemble unseen ones. All results are in simulation, and the paper is a submission to ICLR 2027, not yet peer reviewed. Watch for real-robot tests and for the release of code for the OGPO method.
Key facts
- Post-training scaled from 1 to 144 training tasks in MolmoSpaces
- Critic-guided methods: BoN and OGPO; filtered supervised fine-tuning (FSFT) also tested
- Gains over the base policy up to +58 pp on unseen scenes and +39 pp on unseen objects
- Single-task post-training degraded performance on other tasks
- Authors from ETH Zurich, UC Berkeley, CMU, TUM, Max Planck Institute for Intelligent Systems, University of Warsaw and Nomagic
- Marked as an ICLR 2027 submission