End-to-End Test-Time Training for Long Context
We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.
6954c9c9869a8627b452c99b
Arnuv Tandon
6954c9c9869a8627b452c99c
Karan Dalal
6954c9c9869a8627b452c99d
Xinhao Li
6954c9c9869a8627b452c99e
Daniel Koceja
6954c9c9869a8627b452c99f
Marcel Rød
6954c9c9869a8627b452c9a0
Sam Buchanan
6954c9c9869a8627b452c9a1
Xiaolong Wang
6954c9c9869a8627b452c9a2
Jure Leskovec
6954c9c9869a8627b452c9a3
Sanmi Koyejo
6954c9c9869a8627b452c9a4
Tatsunori Hashimoto
6954c9c9869a8627b452c9a5
Carlos Guestrin
6954c9c9869a8627b452c9a6
Jed McCaleb
6954c9c9869a8627b452c9a7
Yejin Choi
6954c9c9869a8627b452c9a8
Yu Sun
End-to-End Test-Time Training for Long Context
We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.