PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
RL post-training with verifiable rewards (RLVR) has become a practical route to eliciting chain-of-thought reasoning in vision--language models (VLMs), but scaling it in the visual domain remains challenging due to costly or noisy supervision and reliance on external verifiers. Puzzle-based RLVR is a promising alternative, yet existing approaches often treat puzzle rewards as flat or sparse, which weakens group-relative learning signal. Existing curriculum strategies are overly restrictive: they rely mainly on reward statistics and do not account for exploration in the solution space, which can lead to collapsed rollout dynamics. Further, RL post-training can induce reasoning--answer inconsistency as training progresses. To address these shortcomings, we present PuzzleCraft, a supervision-free framework that scales vision-centric RLVR using a set of lightweight puzzle environments with built-in verification. PuzzleCraft instantiates three puzzles inspired by classic visual pretext tasks: PatchFit, Rotation, and Jigsaw. We introduce a curriculum that combines difficulty with an exploration signal derived from solution-space dispersion, and use it to downweight collapsed prompt groups. In addition, we introduce a new post-training metric, Reasoning-Answer Consistency (RAC), to measure the degree that the chain-of-though supports the answer, and show our exploration-aware curriculum improves RAC and downstream performance. Across a broad suite of vision-centric benchmarks, PuzzleCraft improves robustness and reasoning consistency, yielding consistent downstream gains on both Qwen2.5-VL and Qwen3-VL backbones. Overall, our results suggest that scalable puzzle-based RLVR benefits from curricula that account for both difficulty and solution-space collapse, together with explicit consistency-enhancing schemes.
6943feea47055eb55aade03e
Ahmadreza Jeddi
claimed_verified
2025-12-19T09:06:56.770Z
65e8df170cda621164769f6f
/avatars/b5d7c7c49d1a777a1bd41d06d299868a.svg
Armen Jeddi
armenjeddi
user
armenjeddi
6943feea47055eb55aade03f
Hakki Can Karaimer
claimed_verified
2025-12-19T09:06:53.951Z
689e4a5b555fe48d3448950f
https://cdn-avatars.huggingface.co/v1/production/uploads/689e4a5b555fe48d3448950f/lFT_s5j2gxSltAXqHtOrk.png
Hakki Karaimer
karaimer
user
karaimer
6943feea47055eb55aade040
Hue Nguyen
6943feea47055eb55aade041
Zhongling Wang
6943feea47055eb55aade042
Ke Zhao
claimed_verified
2026-01-04T20:32:10.666Z
6571ef7cfc603d4fe2433153
/avatars/fd5ad2c25c22726fa776cfab5c0dc829.svg
Ke Zhao
Jokest
user
Jokest
6943feea47055eb55aade043
Javad Rajabi
6943feea47055eb55aade044
Ran Zhang
6943feea47055eb55aade045
Raghav Goyal
69de7a87d3c84fcbf292145c
Konstantinos G. Derpanis
6943feea47055eb55aade046
Babak Taati
6943feea47055eb55aade047
Radek Grzeszczuk
PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
RL post-training with verifiable rewards (RLVR) has become a practical route to eliciting chain-of-thought reasoning in vision--language models (VLMs), but scaling it in the visual domain remains challenging due to costly or noisy supervision and reliance on external verifiers. Puzzle-based RLVR is a promising alternative, yet existing approaches often treat puzzle rewards as flat or sparse, which weakens group-relative learning signal. Existing curriculum strategies are overly restrictive: they rely mainly on reward statistics and do not account for exploration in the solution space, which can lead to collapsed rollout dynamics. Further, RL post-training can induce reasoning--answer inconsistency as training progresses. To address these shortcomings, we present PuzzleCraft, a supervision-free framework that scales vision-centric RLVR using a set of lightweight puzzle environments with built-in verification. PuzzleCraft instantiates three puzzles inspired by classic visual pretext tasks: PatchFit, Rotation, and Jigsaw. We introduce a curriculum that combines difficulty with an exploration signal derived from solution-space dispersion, and use it to downweight collapsed prompt groups. In addition, we introduce a new post-training metric, Reasoning-Answer Consistency (RAC), to measure the degree that the chain-of-though supports the answer, and show our exploration-aware curriculum improves RAC and downstream performance. Across a broad suite of vision-centric benchmarks, PuzzleCraft improves robustness and reasoning consistency, yielding consistent downstream gains on both Qwen2.5-VL and Qwen3-VL backbones. Overall, our results suggest that scalable puzzle-based RLVR benefits from curricula that account for both difficulty and solution-space collapse, together with explicit consistency-enhancing schemes.
PuzzleCraft presents a supervision-free framework for scaling vision-centric RLVR using lightweight puzzle environments with built-in verification, incorporating an exploration-aware curriculum and a new consistency metric to improve reasoning robustness and downstream performance.
RL post-training vision-language models chain-of-thought reasoning puzzle-based RLVR curriculum learning solution-space dispersion Reasoning-Answer Consistency visual pretext tasks PatchFit Rotation Jigsaw