AT^2PO: Agentic Turn-based Policy Optimization via Tree Search
LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT^2PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT^2PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.
69609fe35b7998385e6395ed
Zefang Zong
admin_assigned
2026-01-09T15:54:07.260Z
64feba7efa64465422ce3003
/avatars/abdc7a3748f6a4e15ebc6aa8d616c87d.svg
zongzefang
zzfoutofspace
user
zzfoutofspace
69609fe35b7998385e6395ee
Dingwei Chen
admin_assigned
2026-01-09T15:54:13.098Z
6462271493f702673bf99c0b
https://cdn-avatars.huggingface.co/v1/production/uploads/6462271493f702673bf99c0b/PyWyI2uoJGr0kpugGGr0t.jpeg
Dingwei Chen
CuSO4-Chen
user
CuSO4-Chen
69609fe35b7998385e6395ef
Yang Li
69609fe35b7998385e6395f0
Qi Yi
69609fe35b7998385e6395f1
Bo Zhou
69609fe35b7998385e6395f2
Chengming Li
admin_assigned
2026-01-09T15:54:28.358Z
65d5f457d032b44853ae79e4
/avatars/362180aff317ecee27513741c18fd98c.svg
chengming li
daming8000
user
daming8000
69609fe35b7998385e6395f3
Bo Qian
69609fe35b7998385e6395f4
Peng Chen
admin_assigned
2026-01-09T15:54:22.694Z
60799b15921db717010c7c8e
/avatars/5e33095aa5343538f09831c1ed2230d2.svg
Peng Chen
pengchen
user
pengchen
69609fe35b7998385e6395f5
Jie Jiang
AT^2PO: Agentic Turn-based Policy Optimization via Tree Search
LLM agents have emerged as powerful systems for tackling multi-turn tasks by interleaving internal reasoning and external tool interactions. Agentic Reinforcement Learning has recently drawn significant research attention as a critical post-training paradigm to further refine these capabilities. In this paper, we present AT^2PO (Agentic Turn-based Policy Optimization via Tree Search), a unified framework for multi-turn agentic RL that addresses three core challenges: limited exploration diversity, sparse credit assignment, and misaligned policy optimization. AT^2PO introduces a turn-level tree structure that jointly enables Entropy-Guided Tree Expansion for strategic exploration and Turn-wise Credit Assignment for fine-grained reward propagation from sparse outcomes. Complementing this, we propose Agentic Turn-based Policy Optimization, a turn-level learning objective that aligns policy updates with the natural decision granularity of agentic interactions. ATPO is orthogonal to tree search and can be readily integrated into any multi-turn RL pipeline. Experiments across seven benchmarks demonstrate consistent improvements over the state-of-the-art baseline by up to 1.84 percentage points in average, with ablation studies validating the effectiveness of each component. Our code is available at https://github.com/zzfoutofspace/ATPO.
AT²PO is a unified framework for multi-turn agentic reinforcement learning that improves exploration diversity, credit assignment, and policy optimization through tree search and turn-level learning objectives.
Agentic Reinforcement Learning tree search Entropy-Guided Tree Expansion Turn-wise Credit Assignment Agentic Turn-based Policy Optimization multi-turn tasks policy optimization reward propagation turn-level learning objective