Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks.
69255d6f16eb3a9f13103aa3
Haoyang Hong
69255d6f16eb3a9f13103aa4
Jiajun Yin
69255d6f16eb3a9f13103aa5
Yuan Wang
69255d6f16eb3a9f13103aa6
Jingnan Liu
69255d6f16eb3a9f13103aa7
Zhe Chen
69255d6f16eb3a9f13103aa8
Ailing Yu
69255d6f16eb3a9f13103aa9
Ji Li
69255d6f16eb3a9f13103aaa
Zhiling Ye
69255d6f16eb3a9f13103aab
Hansong Xiao
69255d6f16eb3a9f13103aac
Yefei Chen
69255d6f16eb3a9f13103aad
Hualei Zhou
69255d6f16eb3a9f13103aae
Yun Yue
69255d6f16eb3a9f13103aaf
Minghui Yang
69255d6f16eb3a9f13103ab0
Chunxiao Guo
69255d6f16eb3a9f13103ab1
Junwei Liu
69255d6f16eb3a9f13103ab2
Peng Wei
69255d6f16eb3a9f13103ab3
Jinjie Gu
Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO
Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks.
M-GRPO, an extension of Group Relative Policy Optimization for hierarchical multi-agent systems, improves stability and efficiency in tool-augmented reasoning tasks by aligning heterogeneous trajectories and decoupling agent training.
multi-agent systems large language model (LLM) Group Relative Policy Optimization hierarchical credit assignment trajectory-alignment scheme decoupled training pipeline shared store cross-server backpropagation tool-augmented reasoning tasks