MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching
Tool-Integrated Reasoning (TIR) empowers large language models (LLMs) to tackle complex tasks by interleaving reasoning steps with external tool interactions. However, existing reinforcement learning methods typically rely on outcome- or trajectory-level rewards, assigning uniform advantages to all steps within a trajectory. This coarse-grained credit assignment fails to distinguish effective tool calls from redundant or erroneous ones, particularly in long-horizon multi-turn scenarios. To address this, we propose MatchTIR, a framework that introduces fine-grained supervision via bipartite matching-based turn-level reward assignment and dual-level advantage estimation. Specifically, we formulate credit assignment as a bipartite matching problem between predicted and ground-truth traces, utilizing two assignment strategies to derive dense turn-level rewards. Furthermore, to balance local step precision with global task success, we introduce a dual-level advantage estimation scheme that integrates turn-level and trajectory-level signals, assigning distinct advantage values to individual interaction turns. Extensive experiments on three benchmarks demonstrate the superiority of MatchTIR. Notably, our 4B model surpasses the majority of 8B competitors, particularly in long-horizon and multi-turn tasks. Our codes are available at https://github.com/quchangle1/MatchTIR.
6969b31632f0333869ff948f
Changle Qu
claimed_verified
2026-01-16T10:29:29.887Z
6640c4b0b2f118e6197e12d7
https://cdn-avatars.huggingface.co/v1/production/uploads/6640c4b0b2f118e6197e12d7/I-4leoFMfs4fD0J-5Uuy6.jpeg
changle_qu
ChangleQu
user
ChangleQu
6969b31632f0333869ff9490
Sunhao Dai
claimed_verified
2026-01-22T08:47:34.469Z
64db88993725f8d9a908c077
https://cdn-avatars.huggingface.co/v1/production/uploads/64db88993725f8d9a908c077/JZEGk0kius6mrANlwOWw9.jpeg
Sunhao Dai
KID-22
user
KID-22
6969b31632f0333869ff9491
Hengyi Cai
6969b31632f0333869ff9492
Jun Xu
6969b31632f0333869ff9493
Shuaiqiang Wang
6969b31632f0333869ff9494
Dawei Yin
MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching
Tool-Integrated Reasoning (TIR) empowers large language models (LLMs) to tackle complex tasks by interleaving reasoning steps with external tool interactions. However, existing reinforcement learning methods typically rely on outcome- or trajectory-level rewards, assigning uniform advantages to all steps within a trajectory. This coarse-grained credit assignment fails to distinguish effective tool calls from redundant or erroneous ones, particularly in long-horizon multi-turn scenarios. To address this, we propose MatchTIR, a framework that introduces fine-grained supervision via bipartite matching-based turn-level reward assignment and dual-level advantage estimation. Specifically, we formulate credit assignment as a bipartite matching problem between predicted and ground-truth traces, utilizing two assignment strategies to derive dense turn-level rewards. Furthermore, to balance local step precision with global task success, we introduce a dual-level advantage estimation scheme that integrates turn-level and trajectory-level signals, assigning distinct advantage values to individual interaction turns. Extensive experiments on three benchmarks demonstrate the superiority of MatchTIR. Notably, our 4B model surpasses the majority of 8B competitors, particularly in long-horizon and multi-turn tasks. Our codes are available at https://github.com/quchangle1/MatchTIR.
MatchTIR enhances LLM reasoning by introducing fine-grained credit assignment through bipartite matching and dual-level advantage estimation for tool-integrated tasks.
tool-integrated reasoning reinforcement learning credit assignment bipartite matching turn-level reward assignment dual-level advantage estimation trajectory-level signals long-horizon tasks multi-turn scenarios