Test-Time Training with KV Binding Is Secretly Linear Attention
Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity.
699e6e60dfbcf0b800aecb16
Junchen Liu
699e6e60dfbcf0b800aecb17
Sven Elflein
claimed_verified
2026-02-27T16:43:38.764Z
692da82ab884a1ba56270557
/avatars/e98409da85032ad2092b77f9fceee784.svg
Sven Elflein
sven-el
user
sven-el
699e6e60dfbcf0b800aecb18
Or Litany
699e6e60dfbcf0b800aecb19
Zan Gojcic
699e6e60dfbcf0b800aecb1a
Ruilong Li
Test-Time Training with KV Binding Is Secretly Linear Attention
Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity.
Test-time training is reinterpreted as learned linear attention rather than memorization, offering architectural simplifications and improved efficiency.
test-time training KV binding online meta-learning learned linear attention sequence modeling layer linear attention operator architectural simplifications parallel formulations representational capacity