Tool Verification for Test-Time Reinforcement Learning
Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for self-evolving large reasoning models (LRMs), enabling online adaptation on unlabeled test inputs via self-induced rewards through majority voting. However, a spurious yet high-frequency unverified consensus can become a biased and reinforced reward signal, leading to incorrect mode collapse. We address this failure mode with T^3RL (Tool-Verification for Test-Time Reinforcement Learning), which introduces test-time tool verification into reward estimation. Concretely, a verifier uses an external tool as evidence (e.g., from code execution) to upweight verified rollouts in a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone types, T^3RL significantly improves over TTRL, with larger gains on harder problems. More broadly, T^3RL can be viewed as verified online data synthesis, highlighting test-time tool verification as a key mechanism for stabilizing self-evolution.
69a66b66663a122a7ce667bf
Ruotong Liao
claimed_verified
2026-03-09T10:38:17.894Z
642d941691aa2e3597e7bad8
/avatars/96c69e0160613fa5b50ee73e244c004c.svg
Ruotong Liao
mayhugo99
user
mayhugo99
69a66b66663a122a7ce667c0
Nikolai Röhrich
69a66b66663a122a7ce667c1
Xiaohan Wang
69a66b66663a122a7ce667c2
Yuhui Zhang
69a66b66663a122a7ce667c3
Yasaman Samadzadeh
69a66b66663a122a7ce667c4
Volker Tresp
69a66b66663a122a7ce667c5
Serena Yeung-Levy
Tool Verification for Test-Time Reinforcement Learning
Test-time reinforcement learning (TTRL) has emerged as a promising paradigm for self-evolving large reasoning models (LRMs), enabling online adaptation on unlabeled test inputs via self-induced rewards through majority voting. However, a spurious yet high-frequency unverified consensus can become a biased and reinforced reward signal, leading to incorrect mode collapse. We address this failure mode with T^3RL (Tool-Verification for Test-Time Reinforcement Learning), which introduces test-time tool verification into reward estimation. Concretely, a verifier uses an external tool as evidence (e.g., from code execution) to upweight verified rollouts in a verification-aware voting, producing more reliable pseudo-labels for training. Across various math difficulties (MATH-500, AMC, and AIME 2024) and diverse backbone types, T^3RL significantly improves over TTRL, with larger gains on harder problems. More broadly, T^3RL can be viewed as verified online data synthesis, highlighting test-time tool verification as a key mechanism for stabilizing self-evolution.
Test-time reinforcement learning with tool verification addresses consensus bias in large reasoning models by using external validation to improve reward estimation and model stability.
test-time reinforcement learning large reasoning models self-evolving models majority voting mode collapse tool verification reward estimation pseudo-labels online data synthesis