Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems
Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6\% avg@16 and +4.6\% pass@16 on math, and +15.2\% avg@16 and +13.1\% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.
698c133d6052d3bed9630bf2
Lang Feng
claimed_verified
2026-02-11T11:13:38.406Z
66ba29dd59e8e7a957154c5f
https://cdn-avatars.huggingface.co/v1/production/uploads/66ba29dd59e8e7a957154c5f/VvVS7IZNPUIB023GAEf5u.png
Lang Feng
langfeng01
user
langfeng01
698c133d6052d3bed9630bf3
Longtao Zheng
claimed_verified
2026-02-11T11:13:41.494Z
63db5dc49f2687298a1547bf
https://cdn-avatars.huggingface.co/v1/production/uploads/63db5dc49f2687298a1547bf/xVFi0kRkYud191cQgma16.jpeg
Longtao Zheng
ltzheng
user
ltzheng
698c133d6052d3bed9630bf4
Shuo He
claimed_verified
2026-02-12T13:57:44.843Z
641d6099f9a3a9c532bd3954
https://cdn-avatars.huggingface.co/v1/production/uploads/641d6099f9a3a9c532bd3954/NBrTueSRKkxhdLvCdZ0lt.jpeg
Shuo He
heshuo
user
heshuo
698c133d6052d3bed9630bf5
Fuxiang Zhang
claimed_verified
2026-02-11T11:13:54.465Z
64054d8a3d49e1e066bfa32b
/avatars/9044f937145cc5aa4bc3a5ffa751f724.svg
Fuxiang Zhang
sicer
user
sicer
698c133d6052d3bed9630bf6
Bo An
Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems
Multi-agent LLM systems enable advanced reasoning and tool use via role specialization, yet reliable reinforcement learning (RL) post-training for such systems remains difficult. In this work, we theoretically pinpoint a key reason for training instability when extending group-based RL to multi-agent LLM systems. We show that under GRPO-style optimization, a global normalization baseline may deviate from diverse agents' reward distributions, which ultimately leads to gradient-norm instability. Based on this finding, we propose Dr. MAS, a simple and stable RL training recipe for multi-agent LLM systems. Dr. MAS uses an agent-wise remedy: normalizing advantages per agent using each agent's own reward statistics, which calibrates gradient scales and dramatically stabilizes training, both theoretically and empirically. Beyond the algorithm, Dr. MAS provides an end-to-end RL training framework for multi-agent LLM systems, supporting scalable orchestration, flexible per-agent LLM serving and optimization configs, and shared resource scheduling of LLM actor backends. We evaluate Dr. MAS on multi-agent math reasoning and multi-turn search benchmarks using Qwen2.5 and Qwen3 series models. Dr. MAS achieves clear gains over vanilla GRPO (e.g., +5.6\% avg@16 and +4.6\% pass@16 on math, and +15.2\% avg@16 and +13.1\% pass@16 on search) while largely eliminating gradient spikes. Moreover, it remains highly effective under heterogeneous agent-model assignments while improving efficiency.
Multi-agent large language model systems face training instability in reinforcement learning due to global normalization mismatches, which is addressed by Dr. MAS through agent-specific advantage normalization and enhanced training stability.
multi-agent LLM systems reinforcement learning GRPO-style optimization gradient-norm instability agent-wise remedy advantage normalization Qwen2.5 Qwen3 series