Heterogeneous Agent Collaborative Reinforcement Learning
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new learning paradigm that addresses the inefficiencies of isolated on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional teacher-to-student transfer. Building on this paradigm, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation and optimization correctness. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO by an average of 3.3\% while using only half the rollout cost.
69a8ec8e8b5b27c7ea9fa5c5
Zhixia Zhang
admin_assigned
2026-03-05T10:25:12.867Z
68a7338193838aa4c56fcafc
/avatars/b33dbbc3f020bd399085e8e41ee3a839.svg
Zhixia Zhang
zzx-peter
user
zzx-peter
69a8ec8e8b5b27c7ea9fa5c6
Zixuan Huang
claimed_verified
2026-03-05T10:01:21.690Z
6593d2329e16fa7510e0876a
/avatars/c200676dd9dc1db8a3e27388251aea49.svg
hzx
hzxllll
user
hzxllll
69a8ec8e8b5b27c7ea9fa5c7
Xin Xia
69a8ec8e8b5b27c7ea9fa5c8
Deqing Wang
69a8ec8e8b5b27c7ea9fa5c9
Fuzhen Zhuang
69a8ec8e8b5b27c7ea9fa5ca
Shuai Ma
69a8ec8e8b5b27c7ea9fa5cb
Ning Ding
69a8ec8e8b5b27c7ea9fa5cc
Yaodong Yang
claimed_verified
2026-03-06T13:56:48.607Z
686894eb283604c2808f7c81
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/BGuMtSS3BGNicQiojujUO.png
yd yang
yyang13
user
yyang13
69a8ec8e8b5b27c7ea9fa5cd
Jianxin Li
admin_assigned
2026-03-05T10:25:53.231Z
6434f9c4df32a2296635f8af
/avatars/edfd074b565d8d7d61579a3202d2ab71.svg
jianxinli
jianxinli
user
jianxinli
69a8ec8e8b5b27c7ea9fa5ce
Yikun Ban
claimed_verified
2026-03-06T13:56:51.497Z
68345345f4bbf856e2d708e2
https://cdn-avatars.huggingface.co/v1/production/uploads/68345345f4bbf856e2d708e2/L5H2HNCuWje3ti2tNbC5p.jpeg
Yikun Ban
Yikunb
user
Yikunb
Heterogeneous Agent Collaborative Reinforcement Learning
We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new learning paradigm that addresses the inefficiencies of isolated on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional teacher-to-student transfer. Building on this paradigm, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation and optimization correctness. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO by an average of 3.3\% while using only half the rollout cost.
HACRL enables collaborative reinforcement learning where heterogeneous agents share verified rollouts during training to improve collectively while maintaining independent operation at inference time, with HACPO achieving superior performance through efficient sample utilization and cross-agent knowledge transfer.
heterogeneous agents collaborative optimization on-policy optimization multi-agent reinforcement learning rollout sharing sample utilization cross-agent knowledge transfer advantage estimation policy distribution shifts