RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via
  Hierarchical Model Merging
We unveil that internal representations in large language models (LLMs) serve
as reliable proxies of learned knowledge, and propose RECALL, a novel
representation-aware model merging framework for continual learning without
access to historical data. RECALL computes inter-model similarity from
layer-wise hidden representations over clustered typical samples, and performs
adaptive, hierarchical parameter fusion to align knowledge across models. This
design enables the preservation of domain-general features in shallow layers
while allowing task-specific adaptation in deeper layers. Unlike prior methods
that require task labels or incur performance trade-offs, RECALL achieves
seamless multi-domain integration and strong resistance to catastrophic
forgetting. Extensive experiments across five NLP tasks and multiple continual
learning scenarios show that RECALL outperforms baselines in both knowledge
retention and generalization, providing a scalable and data-free solution for
evolving LLMs.
68fb97d46cdff8b857f46ba2
Bowen Wang
claimed_verified
2025-10-27T10:28:59.550Z
66e6e89b3fbe08f0a3a07066
/avatars/c76cffd761422d3cf999d94b48423f9d.svg
Makise Kurisu
Makise-Kurisu
user
Makise-Kurisu
68fb97d46cdff8b857f46ba3
Haiyuan Wan
68fb97d46cdff8b857f46ba4
Liwen Shi
68fb97d46cdff8b857f46ba5
Chen Yang
68fb97d46cdff8b857f46ba6
Peng He
68fb97d46cdff8b857f46ba7
Yue Ma
68fb97d46cdff8b857f46ba8
Haochen Han
68fb97d46cdff8b857f46ba9
Wenhao Li
68fb97d46cdff8b857f46baa
Tiao Tan
68fb97d46cdff8b857f46bab
Yongjian Li
68fb97d46cdff8b857f46bac
Fangming Liu
68fb97d46cdff8b857f46bad
Yifan Gong
68fb97d46cdff8b857f46bae
Sheng Zhang
RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via
  Hierarchical Model Merging
We unveil that internal representations in large language models (LLMs) serve
as reliable proxies of learned knowledge, and propose RECALL, a novel
representation-aware model merging framework for continual learning without
access to historical data. RECALL computes inter-model similarity from
layer-wise hidden representations over clustered typical samples, and performs
adaptive, hierarchical parameter fusion to align knowledge across models. This
design enables the preservation of domain-general features in shallow layers
while allowing task-specific adaptation in deeper layers. Unlike prior methods
that require task labels or incur performance trade-offs, RECALL achieves
seamless multi-domain integration and strong resistance to catastrophic
forgetting. Extensive experiments across five NLP tasks and multiple continual
learning scenarios show that RECALL outperforms baselines in both knowledge
retention and generalization, providing a scalable and data-free solution for
evolving LLMs.
RECALL is a representation-aware framework for continual learning in large language models that merges models without historical data, preserving domain-general features and adapting to task-specific knowledge.
large language models LLMs RECALL representation-aware continual learning inter-model similarity layer-wise hidden representations adaptive hierarchical parameter fusion domain-general features task-specific adaptation catastrophic forgetting NLP tasks