Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories
In this paper, we aim to bridge test-time-training with a new type of parametric memory that can be flexibly offloaded from or merged into model parameters. We present Locas, a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers, allowing it to be flexibly permanentized into the model parameters while supporting efficient continual learning. We discuss two major variants of Locas: one with a conventional two-layer MLP design that has a clearer theoretical guarantee; the other one shares the same GLU-FFN structure with SOTA LLMs, and can be easily attached to existing models for both parameter-efficient and computation-efficient continual learning. Crucially, we show that proper initialization of such low-rank sideway-FFN-style memories -- performed in a principled way by reusing model parameters, activations and/or gradients -- is essential for fast convergence, improved generalization, and catastrophic forgetting prevention. We validate the proposed memory mechanism on the PG-19 whole-book language modeling and LoCoMo long-context dialogue question answering tasks. With only 0.02\% additional parameters in the lowest case, Locas-GLU is capable of storing the information from past context while maintaining a much smaller context window. In addition, we also test the model's general capability loss after memorizing the whole book with Locas, through comparative MMLU evaluation. Results show the promising ability of Locas to permanentize past context into parametric knowledge with minimized catastrophic forgetting of the model's existing internal knowledge.
698d062b6c5152984e4f3e27
Sidi Lu
claimed_verified
2026-02-12T13:57:18.928Z
65f2a015f79caa96bcde9de4
/avatars/b989383d43162a42e264001ff940ea73.svg
Sidi Lu
desire2020
user
desire2020
698d062b6c5152984e4f3e28
Zhenwen Liang
698d062b6c5152984e4f3e29
Dongyang Ma
698d062b6c5152984e4f3e2a
Yan Wang
claimed_verified
2026-02-12T13:57:16.938Z
64674375946476c5d215382d
/avatars/0f0b98080c064a17d003da8a22a047cc.svg
Yan Wang
libertywing
user
libertywing
698d062b6c5152984e4f3e2b
Haitao Mi
698d062b6c5152984e4f3e2c
Dong Yu
Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories
In this paper, we aim to bridge test-time-training with a new type of parametric memory that can be flexibly offloaded from or merged into model parameters. We present Locas, a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers, allowing it to be flexibly permanentized into the model parameters while supporting efficient continual learning. We discuss two major variants of Locas: one with a conventional two-layer MLP design that has a clearer theoretical guarantee; the other one shares the same GLU-FFN structure with SOTA LLMs, and can be easily attached to existing models for both parameter-efficient and computation-efficient continual learning. Crucially, we show that proper initialization of such low-rank sideway-FFN-style memories -- performed in a principled way by reusing model parameters, activations and/or gradients -- is essential for fast convergence, improved generalization, and catastrophic forgetting prevention. We validate the proposed memory mechanism on the PG-19 whole-book language modeling and LoCoMo long-context dialogue question answering tasks. With only 0.02\% additional parameters in the lowest case, Locas-GLU is capable of storing the information from past context while maintaining a much smaller context window. In addition, we also test the model's general capability loss after memorizing the whole book with Locas, through comparative MMLU evaluation. Results show the promising ability of Locas to permanentize past context into parametric knowledge with minimized catastrophic forgetting of the model's existing internal knowledge.
Locas, a locally-supported parametric memory mechanism, enables flexible integration with transformer models for continual learning while minimizing catastrophic forgetting through principled initialization techniques.
parametric memory transformer FFN blocks continual learning low-rank sideway-FFN-style memories GLU-FFN parameter-efficient learning computation-efficient learning catastrophic forgetting MMLU evaluation