Higher-order Linear Attention
The quadratic cost of scaled dot-product attention is a central obstacle to
scaling autoregressive language models to long contexts. Linear-time attention
and State Space Models (SSMs) provide scalable alternatives but are typically
restricted to first-order or kernel-based approximations, which can limit
expressivity. We introduce Higher-order Linear Attention (HLA), a causal,
streaming mechanism that realizes higher interactions via compact prefix
sufficient statistics. In the second-order case, HLA maintains a constant-size
state and computes per-token outputs in linear time without materializing any
n times n matrices. We give closed-form streaming identities, a strictly
causal masked variant using two additional summaries, and a chunk-parallel
training scheme based on associative scans that reproduces the activations of a
serial recurrence exactly. We further outline extensions to third and higher
orders. Collectively, these results position HLA as a principled, scalable
building block that combines attention-like, data-dependent mixing with the
efficiency of modern recurrent architectures. Project Page:
https://github.com/yifanzhang-pro/HLA.
69081a51812eca10f9cc5e5c
Yifan Zhang
claimed_verified
2025-11-03T20:52:21.886Z
647bf082aba7062fe5c51ca9
https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg
Yifan Zhang
yifAI
user
yifAI
69081a51812eca10f9cc5e5d
Zhen Qin
69081a51812eca10f9cc5e5e
Quanquan Gu
Higher-order Linear Attention
The quadratic cost of scaled dot-product attention is a central obstacle to
scaling autoregressive language models to long contexts. Linear-time attention
and State Space Models (SSMs) provide scalable alternatives but are typically
restricted to first-order or kernel-based approximations, which can limit
expressivity. We introduce Higher-order Linear Attention (HLA), a causal,
streaming mechanism that realizes higher interactions via compact prefix
sufficient statistics. In the second-order case, HLA maintains a constant-size
state and computes per-token outputs in linear time without materializing any
n times n matrices. We give closed-form streaming identities, a strictly
causal masked variant using two additional summaries, and a chunk-parallel
training scheme based on associative scans that reproduces the activations of a
serial recurrence exactly. We further outline extensions to third and higher
orders. Collectively, these results position HLA as a principled, scalable
building block that combines attention-like, data-dependent mixing with the
efficiency of modern recurrent architectures. Project Page:
https://github.com/yifanzhang-pro/HLA.
Higher-order Linear Attention (HLA) is a scalable, causal, and efficient mechanism for long-context autoregressive language models, combining attention-like mixing with recurrent architecture efficiency.
scaled dot-product attention linear-time attention State Space Models (SSMs) Higher-order Linear Attention (HLA) causal streaming mechanism prefix sufficient statistics closed-form streaming identities chunk-parallel training scheme associative scans serial recurrence