OpenSIR: Open-Ended Self-Improving Reasoner
Recent advances in large language model (LLM) reasoning through reinforcement
learning rely on annotated datasets for verifiable rewards, which may limit
models' ability to surpass human-level performance. While self-play offers a
promising alternative, existing approaches depend on external verifiers or
cannot learn open-endedly. We present Open-Ended Self-Improving Reasoner
(OpenSIR), a self-play framework where an LLM learns to generate and solve
novel problems by alternating teacher and student roles without external
supervision. To generate novel problems, OpenSIR optimises for both difficulty
and diversity, rewarding problems that challenge appropriately while exploring
distinct concepts, enabling open-ended mathematical discovery. Starting from a
single trivial seed problem, OpenSIR substantially improves instruction models:
Llama-3.2-3B-Instruct advances from 73.9 to 78.3 on GSM8K, and from 28.8 to
34.4 on College Math, while Gemma-2-2B-Instruct rises from 38.5 to 58.7 on
GSM8K. Our analyses reveal that OpenSIR achieves open-ended learning through
co-evolving teacher-student roles that adaptively calibrate difficulty and
drive diverse exploration, progressing autonomously from basic to advanced
mathematics.
6909a672812eca10f9cc63f7
Wai-Chung Kwan
6909a672812eca10f9cc63f8
Joshua Ong Jun Leang
6909a672812eca10f9cc63f9
Pavlos Vougiouklis
6909a672812eca10f9cc63fa
Jeff Z. Pan
6909a672812eca10f9cc63fb
Marco Valentino
6909a672812eca10f9cc63fc
Pasquale Minervini
claimed_verified
2025-11-04T09:24:52.430Z
61001311e043e15c13412d30
https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
Pasquale Minervini
pminervini
user
pminervini
OpenSIR: Open-Ended Self-Improving Reasoner
Recent advances in large language model (LLM) reasoning through reinforcement
learning rely on annotated datasets for verifiable rewards, which may limit
models' ability to surpass human-level performance. While self-play offers a
promising alternative, existing approaches depend on external verifiers or
cannot learn open-endedly. We present Open-Ended Self-Improving Reasoner
(OpenSIR), a self-play framework where an LLM learns to generate and solve
novel problems by alternating teacher and student roles without external
supervision. To generate novel problems, OpenSIR optimises for both difficulty
and diversity, rewarding problems that challenge appropriately while exploring
distinct concepts, enabling open-ended mathematical discovery. Starting from a
single trivial seed problem, OpenSIR substantially improves instruction models:
Llama-3.2-3B-Instruct advances from 73.9 to 78.3 on GSM8K, and from 28.8 to
34.4 on College Math, while Gemma-2-2B-Instruct rises from 38.5 to 58.7 on
GSM8K. Our analyses reveal that OpenSIR achieves open-ended learning through
co-evolving teacher-student roles that adaptively calibrate difficulty and
drive diverse exploration, progressing autonomously from basic to advanced
mathematics.
OpenSIR is a self-play framework that enables large language models to improve their reasoning abilities through open-ended problem generation and solving without external supervision.
large language model reinforcement learning annotated datasets self-play Open-Ended Self-Improving Reasoner OpenSIR teacher-student roles open-ended learning mathematical discovery GSM8K College Math Gemma-2-2B-Instruct Llama-3.2-3B-Instruct