SageBwd: A Trainable Low-bit Attention
Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In prior work, we introduced SageBwd, a trainable INT8 attention that quantizes six of seven attention matrix multiplications while preserving fine-tuning performance. However, SageBwd exhibited a persistent performance gap to full-precision attention (FPA) during pre-training. In this work, we investigate why this gap occurs and demonstrate that SageBwd matches full-precision attention during pretraining. Through experiments and theoretical analysis, we reach a few important insights and conclusions: (i) QK-norm is necessary for stable training at large tokens per step, (ii) quantization errors primarily arise from the backward-pass score gradient dS, (iii) reducing tokens per step enables SageBwd to match FPA performance in pre-training, and (iv) K-smoothing remains essential for training stability, while Q-smoothing provides limited benefit during pre-training.
69aa5208c4f6edddcc6c86e8
Jintao Zhang
claimed_verified
2026-03-06T13:55:00.258Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
jt-zhang
user
jt-zhang
69aa5208c4f6edddcc6c86e9
Marco Chen
69aa5208c4f6edddcc6c86ea
Haoxu Wang
69aa5208c4f6edddcc6c86eb
Kai Jiang
69aa5208c4f6edddcc6c86ec
Ion Stoica
69aa5208c4f6edddcc6c86ed
Joseph E. Gonzalez
69aa5208c4f6edddcc6c86ee
Jianfei Chen
69aa5208c4f6edddcc6c86ef
Jun Zhu
SageBwd: A Trainable Low-bit Attention
Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In prior work, we introduced SageBwd, a trainable INT8 attention that quantizes six of seven attention matrix multiplications while preserving fine-tuning performance. However, SageBwd exhibited a persistent performance gap to full-precision attention (FPA) during pre-training. In this work, we investigate why this gap occurs and demonstrate that SageBwd matches full-precision attention during pretraining. Through experiments and theoretical analysis, we reach a few important insights and conclusions: (i) QK-norm is necessary for stable training at large tokens per step, (ii) quantization errors primarily arise from the backward-pass score gradient dS, (iii) reducing tokens per step enables SageBwd to match FPA performance in pre-training, and (iv) K-smoothing remains essential for training stability, while Q-smoothing provides limited benefit during pre-training.
Research investigates why low-bit attention methods like SageBwd exhibit performance gaps during pre-training and identifies key factors for stable training with reduced precision.
SageAttention INT8 attention attention matrix multiplications fine-tuning performance full-precision attention QK-norm quantization errors backward-pass score gradient tokens per step K-smoothing Q-smoothing