ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
Reward models are critical for aligning vision-language systems with human preferences, yet current approaches suffer from hallucination, weak visual grounding, and an inability to use tools for verification, limiting their reliability on complex multimodal reasoning tasks. We present ARM-Thinker, an A}gentic multimodal Reward Model that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence, replacing static, non-interactive reward scoring. This enables the model to verify fine-grained visual details, cross-reference multi-page evidence, and validate reasoning claims, which are capabilities absent in existing reward models. We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy. To evaluate agentic reward modeling, we introduce ARMBench-VL, comprising three benchmarks that assess fine-grained visual grounding (image-level tools), multi-page document understanding (retrieval tools), and instruction following (text-level verification). ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms baselines on multimodal math and logical reasoning benchmarks. Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.
693267756d1060ca587a2780
Shengyuan Ding
claimed_verified
2025-12-09T09:38:36.881Z
646cd947da8e99940b6e55cf
https://cdn-avatars.huggingface.co/v1/production/uploads/646cd947da8e99940b6e55cf/9c0P0WppFqNW9pdo8LgOS.jpeg
Shengyuan Ding
ChrisDing1105
user
ChrisDing1105
693267756d1060ca587a2781
Xinyu Fang
claimed_verified
2025-12-05T08:39:08.410Z
64f5f8dd9b17cd59c453c57f
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
Xinyu Fang
nebulae09
user
nebulae09
693267756d1060ca587a2782
Ziyu Liu
693267756d1060ca587a2783
Yuhang Zang
claimed_verified
2025-12-05T08:29:11.324Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/kajwuVzd4pDucSPlwghxo.png
Yuhang Zang
yuhangzang
user
yuhangzang
693267756d1060ca587a2784
Yuhang Cao
693267756d1060ca587a2785
Xiangyu Zhao
693267756d1060ca587a2786
Haodong Duan
693267756d1060ca587a2787
Xiaoyi Dong
693267756d1060ca587a2788
Jianze Liang
693267756d1060ca587a2789
Bin Wang
693267756d1060ca587a278a
Conghui He
693267756d1060ca587a278b
Dahua Lin
693267756d1060ca587a278c
Jiaqi Wang
claimed_verified
2026-02-17T15:49:10.071Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
myownskyW7
user
myownskyW7
ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
Reward models are critical for aligning vision-language systems with human preferences, yet current approaches suffer from hallucination, weak visual grounding, and an inability to use tools for verification, limiting their reliability on complex multimodal reasoning tasks. We present ARM-Thinker, an A}gentic multimodal Reward Model that autonomously invokes external tools (e.g., image cropping, doc page retrieval) to ground judgments in verifiable evidence, replacing static, non-interactive reward scoring. This enables the model to verify fine-grained visual details, cross-reference multi-page evidence, and validate reasoning claims, which are capabilities absent in existing reward models. We train ARM-Thinker with multi-stage reinforcement learning, jointly optimizing tool-calling decisions and judgment accuracy. To evaluate agentic reward modeling, we introduce ARMBench-VL, comprising three benchmarks that assess fine-grained visual grounding (image-level tools), multi-page document understanding (retrieval tools), and instruction following (text-level verification). ARM-Thinker achieves +16.2% average improvement on reward modeling benchmarks, +9.6% on tool-use tasks, and outperforms baselines on multimodal math and logical reasoning benchmarks. Our results demonstrate that agentic capabilities significantly enhance both accuracy and interpretability of reward models.
ARM-Thinker, an agentic reward model, uses external tools for verification, improving accuracy and interpretability in complex multimodal reasoning tasks.
reward models vision-language systems hallucination visual grounding external tools image cropping doc page retrieval multi-stage reinforcement learning tool-calling decisions judgment accuracy agentic reward modeling ARMBench-VL fine-grained visual grounding multi-page document understanding instruction following multimodal math logical reasoning