The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Large language models trained on natural language exhibit pronounced anisotropy: a small number of directions concentrate disproportionate energy, while the remaining dimensions form a broad semantic tail. In low-bit training regimes, this geometry becomes numerically unstable. Because blockwise quantization scales are determined by extreme elementwise magnitudes, dominant directions stretch the dynamic range, compressing long-tail semantic variation into narrow numerical bins. We show that this instability is primarily driven by a coherent rank-one mean bias, which constitutes the dominant component of spectral anisotropy in LLM representations. This mean component emerges systematically across layers and training stages and accounts for the majority of extreme activation magnitudes, making it the principal driver of dynamic-range inflation under low precision. Crucially, because the dominant instability is rank-one, it can be eliminated through a simple source-level mean-subtraction operation. This bias-centric conditioning recovers most of the stability benefits of SVD-based spectral methods while requiring only reduction operations and standard quantization kernels. Empirical results on FP4 (W4A4G4) training show that mean removal substantially narrows the loss gap to BF16 and restores downstream performance, providing a hardware-efficient path to stable low-bit LLM training.
69b39f422ad6c5d4cf486b67
Hengjie Cao
69b39f422ad6c5d4cf486b68
Zhendong Huang
69b39f422ad6c5d4cf486b69
Mengyi Chen
69b39f422ad6c5d4cf486b6a
Yifeng Yang
69b39f422ad6c5d4cf486b6b
Fanqi Yu
69b39f422ad6c5d4cf486b6c
Ruijun Huang
69b39f422ad6c5d4cf486b6d
Fang Dong
69b39f422ad6c5d4cf486b6e
Xin Zhang
69b39f422ad6c5d4cf486b6f
Jixian Zhou
69b39f422ad6c5d4cf486b70
Anrui Chen
69b39f422ad6c5d4cf486b71
Mingzhi Dong
69b39f422ad6c5d4cf486b72
Yujiang Wang
69b39f422ad6c5d4cf486b73
Jinlong Hou
69b39f422ad6c5d4cf486b74
Qin Lv
69b39f422ad6c5d4cf486b75
Yuan Cheng
69b39f422ad6c5d4cf486b76
Tun Lu
69b39f422ad6c5d4cf486b77
Fan Yang
69b39f422ad6c5d4cf486b78
Li Shang
The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
Large language models trained on natural language exhibit pronounced anisotropy: a small number of directions concentrate disproportionate energy, while the remaining dimensions form a broad semantic tail. In low-bit training regimes, this geometry becomes numerically unstable. Because blockwise quantization scales are determined by extreme elementwise magnitudes, dominant directions stretch the dynamic range, compressing long-tail semantic variation into narrow numerical bins. We show that this instability is primarily driven by a coherent rank-one mean bias, which constitutes the dominant component of spectral anisotropy in LLM representations. This mean component emerges systematically across layers and training stages and accounts for the majority of extreme activation magnitudes, making it the principal driver of dynamic-range inflation under low precision. Crucially, because the dominant instability is rank-one, it can be eliminated through a simple source-level mean-subtraction operation. This bias-centric conditioning recovers most of the stability benefits of SVD-based spectral methods while requiring only reduction operations and standard quantization kernels. Empirical results on FP4 (W4A4G4) training show that mean removal substantially narrows the loss gap to BF16 and restores downstream performance, providing a hardware-efficient path to stable low-bit LLM training.
LLM anisotropy caused by rank-one mean bias in low-bit training can be stabilized through mean subtraction, recovering performance while enabling efficient hardware deployment.
large language models anisotropy blockwise quantization dynamic range rank-one mean bias spectral anisotropy SVD-based spectral methods mean subtraction FP4 W4A4G4 BF16