Reinforced Fast Weights with Next-Sequence Prediction
Fast weight architectures offer a promising alternative to attention-based transformers for long-context modeling by maintaining constant memory overhead regardless of context length. However, their potential is limited by the next-token prediction (NTP) training paradigm. NTP optimizes single-token predictions and ignores semantic coherence across multiple tokens following a prefix. Consequently, fast weight models, which dynamically update their parameters to store contextual information, learn suboptimal representations that fail to capture long-range dependencies. We introduce REFINE (Reinforced Fast weIghts with Next sEquence prediction), a reinforcement learning framework that trains fast weight models under the next-sequence prediction (NSP) objective. REFINE selects informative token positions based on prediction entropy, generates multi-token rollouts, assigns self-supervised sequence-level rewards, and optimizes the model with group relative policy optimization (GRPO). REFINE is applicable throughout the training lifecycle of pre-trained language models: mid-training, post-training, and test-time training. Our experiments on LaCT-760M and DeltaNet-1.3B demonstrate that REFINE consistently outperforms supervised fine-tuning with NTP across needle-in-a-haystack retrieval, long-context question answering, and diverse tasks in LongBench. REFINE provides an effective and versatile framework for improving long-context modeling in fast weight architectures.
699749f57a658569d5a100a5
Hee Seung Hwang
claimed_verified
2026-02-20T08:36:58.805Z
674e20967cf883ca42da70b0
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wykasfd-MuXV-esBSdbYu.png
Will Hwang
hh2605
user
hh2605
699749f57a658569d5a100a6
Xindi Wu
claimed_verified
2026-02-20T08:36:53.955Z
613940c0905b1938233881e3
https://cdn-avatars.huggingface.co/v1/production/uploads/613940c0905b1938233881e3/Vb4kFWKEq6AILUP9KmCZA.png
Xindi Wu
xindiw
user
xindiw
699749f57a658569d5a100a7
Sanghyuk Chun
claimed_verified
2026-02-20T08:37:01.068Z
6531145022368f78ce8f5a5d
/avatars/dad440aecef695565041c2c57d9f121f.svg
Sanghyuk Chun
SanghyukChun
user
SanghyukChun
699749f57a658569d5a100a8
Olga Russakovsky
Reinforced Fast Weights with Next-Sequence Prediction
Fast weight architectures offer a promising alternative to attention-based transformers for long-context modeling by maintaining constant memory overhead regardless of context length. However, their potential is limited by the next-token prediction (NTP) training paradigm. NTP optimizes single-token predictions and ignores semantic coherence across multiple tokens following a prefix. Consequently, fast weight models, which dynamically update their parameters to store contextual information, learn suboptimal representations that fail to capture long-range dependencies. We introduce REFINE (Reinforced Fast weIghts with Next sEquence prediction), a reinforcement learning framework that trains fast weight models under the next-sequence prediction (NSP) objective. REFINE selects informative token positions based on prediction entropy, generates multi-token rollouts, assigns self-supervised sequence-level rewards, and optimizes the model with group relative policy optimization (GRPO). REFINE is applicable throughout the training lifecycle of pre-trained language models: mid-training, post-training, and test-time training. Our experiments on LaCT-760M and DeltaNet-1.3B demonstrate that REFINE consistently outperforms supervised fine-tuning with NTP across needle-in-a-haystack retrieval, long-context question answering, and diverse tasks in LongBench. REFINE provides an effective and versatile framework for improving long-context modeling in fast weight architectures.
REFINE is a reinforcement learning framework that improves fast weight models for long-context modeling by training under next-sequence prediction instead of next-token prediction, enhancing their ability to capture long-range dependencies.
fast weight architectures attention-based transformers next-token prediction next-sequence prediction reinforcement learning prediction entropy multi-token rollouts self-supervised sequence-level rewards group relative policy optimization needle-in-a-haystack retrieval long-context question answering LongBench