Model Merging with Functional Dual Anchors
Model merging is an efficient post-training strategy for integrating
knowledge from multiple finetuned checkpoints of a shared foundation model.
Existing methods operate in the parameter space, combining task vectors to
mitigate conflicts, but remain constrained by parameter inconsistencies. We
propose Functional Dual Anchors (FDAs), a framework that instead models the
input-representation space. FDAs are synthetic inputs whose induced gradients
align with task vectors, capturing task-specific functional shifts relative to
the pretrained model. This perspective bridges joint multi-task training and
post-hoc merging, offering both robustness and flexibility. We further
introduce a principled initialization scheme and show that FDAs are
complementary to parameter-space model merging. Comprehensive experiments
demonstrate the effectiveness of FDAs in model merging.
68fed3e16cdff8b857f47123
Kexuan Shi
claimed_verified
2025-10-27T10:27:25.298Z
67cff3de91473f9c5ccf0fa8
https://cdn-avatars.huggingface.co/v1/production/uploads/67cff3de91473f9c5ccf0fa8/lyHDTF4foLuOHBPmKc2NH.jpeg
Kexuan Shi
KexuanShi
user
KexuanShi
68fed3e16cdff8b857f47124
Yandong Wen
68fed3e16cdff8b857f47125
Weiyang Liu
Model Merging with Functional Dual Anchors
Model merging is an efficient post-training strategy for integrating
knowledge from multiple finetuned checkpoints of a shared foundation model.
Existing methods operate in the parameter space, combining task vectors to
mitigate conflicts, but remain constrained by parameter inconsistencies. We
propose Functional Dual Anchors (FDAs), a framework that instead models the
input-representation space. FDAs are synthetic inputs whose induced gradients
align with task vectors, capturing task-specific functional shifts relative to
the pretrained model. This perspective bridges joint multi-task training and
post-hoc merging, offering both robustness and flexibility. We further
introduce a principled initialization scheme and show that FDAs are
complementary to parameter-space model merging. Comprehensive experiments
demonstrate the effectiveness of FDAs in model merging.
Functional Dual Anchors (FDAs) enhance model merging by aligning gradients with task vectors in the input-representation space, offering robustness and flexibility compared to parameter-space methods.
Functional Dual Anchors FDAs input-representation space task vectors model merging parameter-space joint multi-task training post-hoc merging