Abstraction as a Memory-Efficient Inductive Bias for Continual Learning
The real world is non-stationary and infinitely complex, requiring intelligent agents to learn continually without the prohibitive cost of retraining from scratch. While online continual learning offers a framework for this setting, learning new information often interferes with previously acquired knowledge, causes forgetting and degraded generalization. To address this, we propose Abstraction-Augmented Training (AAT), a loss-level modification encouraging models to capture the latent relational structure shared across examples. By jointly optimizing over concrete instances and their abstract representations, AAT introduces a memory-efficient inductive bias that stabilizes learning in strictly online data streams, eliminating the need for a replay buffer. To capture the multi-faceted nature of abstraction, we introduce and evaluate AAT on two benchmarks: a controlled relational dataset where abstraction is realized through entity masking, and a narrative dataset where abstraction is expressed through shared proverbs. Our results show that AAT achieves performance comparable to or exceeding strong experience replay (ER) baselines, despite requiring zero additional memory and only minimal changes to the training objective. This work highlights structural abstraction as a powerful, memory-free alternative to ER.
69c19feee9374d7b3df94e67
Elnaz Rahmati
69c19feee9374d7b3df94e68
Nona Ghazizadeh
claimed_verified
2026-03-24T09:22:29.146Z
66f5bfeb1c540729cb300856
/avatars/96510703eb2fefffa55b5a9083e3bf54.svg
Nona Ghazizadeh
nona-ghazizadeh
user
nona-ghazizadeh
69c19feee9374d7b3df94e69
Zhivar Sourati
69c19feee9374d7b3df94e6a
Nina Rouhani
69c19feee9374d7b3df94e6b
Morteza Dehghani
Abstraction as a Memory-Efficient Inductive Bias for Continual Learning
The real world is non-stationary and infinitely complex, requiring intelligent agents to learn continually without the prohibitive cost of retraining from scratch. While online continual learning offers a framework for this setting, learning new information often interferes with previously acquired knowledge, causes forgetting and degraded generalization. To address this, we propose Abstraction-Augmented Training (AAT), a loss-level modification encouraging models to capture the latent relational structure shared across examples. By jointly optimizing over concrete instances and their abstract representations, AAT introduces a memory-efficient inductive bias that stabilizes learning in strictly online data streams, eliminating the need for a replay buffer. To capture the multi-faceted nature of abstraction, we introduce and evaluate AAT on two benchmarks: a controlled relational dataset where abstraction is realized through entity masking, and a narrative dataset where abstraction is expressed through shared proverbs. Our results show that AAT achieves performance comparable to or exceeding strong experience replay (ER) baselines, despite requiring zero additional memory and only minimal changes to the training objective. This work highlights structural abstraction as a powerful, memory-free alternative to ER.
Abstraction-Augmented Training (AAT) enhances online continual learning by capturing latent relational structures through joint optimization of concrete instances and abstract representations, achieving performance comparable to experience replay without memory overhead.
online continual learning experience replay memory-efficient inductive bias relational structure abstract representations entity masking shared proverbs