Behavior Knowledge Merge in Reinforced Agentic Models
Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.
69722bbec1c7409747bf97ed
Xiangchi Yuan
claimed_verified
2026-01-22T17:11:49.186Z
658071c1da07e791d8a6e9dc
/avatars/66348bae88b5945cc54012c52cff6aa2.svg
Xiangchi Yuan
Xiangchi
user
Xiangchi
69722bbec1c7409747bf97ee
Dachuan Shi
69722bbec1c7409747bf97ef
Chunhui Zhang
69722bbec1c7409747bf97f0
Zheyuan Liu
69722bbec1c7409747bf97f1
Shenglong Yao
69722bbec1c7409747bf97f2
Soroush Vosoughi
claimed_verified
2026-04-15T12:10:15.740Z
648169d3cb64a2b85a4606e6
https://cdn-avatars.huggingface.co/v1/production/uploads/648169d3cb64a2b85a4606e6/6nEeE3TWvO9O5dCJTOG0C.jpeg
Soroush Vosoughi
soroushv
user
soroushv
69722bbec1c7409747bf97f3
Wenke Lee
Behavior Knowledge Merge in Reinforced Agentic Models
Reinforcement learning (RL) is central to post-training, particularly for agentic models that require specialized reasoning behaviors. In this setting, model merging offers a practical mechanism for integrating multiple RL-trained agents from different tasks into a single generalist model. However, existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models. The root is a task-vector mismatch between RL and SFT: on-policy RL induces task vectors that are highly sparse and heterogeneous, whereas SFT-style merging implicitly assumes dense and globally comparable task vectors. When standard global averaging is applied under this mismatch, RL's non-overlapping task vectors that encode critical task-specific behaviors are reduced and parameter updates are diluted. To address this issue, we propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models. RAM disentangles shared and task-specific unique parameter updates, averaging shared components while selectively preserving and rescaling unique ones to counteract parameter update dilution. Experiments across multiple agent domains and model architectures demonstrate that RAM not only surpasses merging baselines, but also unlocks synergistic potential among agents to achieve performance superior to that of specialized agents in their domains.
Reinforced Agent Merging (RAM) addresses the limitations of traditional merging methods for reinforcement learning-trained agents by distinguishing shared and task-specific parameters to preserve critical behaviors during model integration.
reinforcement learning model merging agentic models task vectors supervised fine-tuning on-policy RL global averaging parameter updates distribution-aware merging synergistic potential