SPICE: Self-Play In Corpus Environments Improves Reasoning
Self-improving systems require environmental interaction for continuous
adaptation. We introduce SPICE (Self-Play In Corpus Environments), a
reinforcement learning framework where a single model acts in two roles: a
Challenger that mines documents from a large corpus to generate diverse
reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics,
the Challenger creates an automatic curriculum at the frontier of the
Reasoner's capability, while corpus grounding provides the rich,
near-inexhaustible external signal necessary for sustained improvement. Unlike
existing ungrounded self-play methods that offer more limited benefits, SPICE
achieves consistent gains across mathematical (+8.9%) and general reasoning
(+9.8%) benchmarks on multiple model families. Our analysis reveals how
document grounding is a key ingredient in SPICE to continuously generate its
own increasingly challenging goals and achieve them, enabling sustained
self-improvement.
69022f86646208eac0d1f730
Bo Liu
claimed_verified
2025-10-30T14:39:32.340Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
Benjamin-eecs
user
Benjamin-eecs
69022f86646208eac0d1f731
Chuanyang Jin
claimed_verified
2025-10-30T14:39:30.314Z
64beb69801f1983a86a05de2
https://cdn-avatars.huggingface.co/v1/production/uploads/64beb69801f1983a86a05de2/tFyCoqZ6gT8NWkZfuncID.jpeg
Chuanyang Jin
Chuanyang-Jin
user
Chuanyang-Jin
69022f86646208eac0d1f732
Seungone Kim
claimed_verified
2025-12-29T14:25:46.014Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
seungone
user
seungone
69022f86646208eac0d1f733
Weizhe Yuan
69022f86646208eac0d1f734
Wenting Zhao
69022f86646208eac0d1f735
Ilia Kulikov
69022f86646208eac0d1f736
Xian Li
69022f86646208eac0d1f737
Sainbayar Sukhbaatar
69022f86646208eac0d1f738
Jack Lanchantin
claimed_verified
2025-10-30T14:39:34.489Z
65dd30a6ea1a202d3cb469f7
/avatars/14029def026380e6e20822d916b25a72.svg
Jack Lanchantin
jcklcn
user
jcklcn
69022f86646208eac0d1f739
Jason Weston
SPICE: Self-Play In Corpus Environments Improves Reasoning
Self-improving systems require environmental interaction for continuous
adaptation. We introduce SPICE (Self-Play In Corpus Environments), a
reinforcement learning framework where a single model acts in two roles: a
Challenger that mines documents from a large corpus to generate diverse
reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics,
the Challenger creates an automatic curriculum at the frontier of the
Reasoner's capability, while corpus grounding provides the rich,
near-inexhaustible external signal necessary for sustained improvement. Unlike
existing ungrounded self-play methods that offer more limited benefits, SPICE
achieves consistent gains across mathematical (+8.9%) and general reasoning
(+9.8%) benchmarks on multiple model families. Our analysis reveals how
document grounding is a key ingredient in SPICE to continuously generate its
own increasingly challenging goals and achieve them, enabling sustained
self-improvement.
SPICE, a reinforcement learning framework, uses self-play in a corpus environment to continuously improve a model's reasoning capabilities through adversarial dynamics and document grounding.
reinforcement learning self-play Challenger Reasoner adversarial dynamics automatic curriculum corpus grounding mathematical benchmarks general reasoning benchmarks model families self-improvement