Optimizing Large Language Model Training Using FP4 Quantization
The growing computational demands of training large language models (LLMs)
necessitate more efficient methods. Quantized training presents a promising
solution by enabling low-bit arithmetic operations to reduce these costs. While
FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge
due to significant quantization errors and limited representational capacity.
This work introduces the first FP4 training framework for LLMs, addressing
these challenges with two key innovations: a differentiable quantization
estimator for precise weight updates and an outlier clamping and compensation
strategy to prevent activation collapse. To ensure stability, the framework
integrates a mixed-precision training scheme and vector-wise quantization.
Experimental results demonstrate that our FP4 framework achieves accuracy
comparable to BF16 and FP8, with minimal degradation, scaling effectively to
13B-parameter LLMs trained on up to 100B tokens. With the emergence of
next-generation hardware supporting FP4, our framework sets a foundation for
efficient ultra-low precision training.
6799b367d30dc065a2d51592
Ruizhe Wang
claimed_verified
2025-01-30T08:40:31.349Z
63203d4e260e691cfc19fcb1
/avatars/72437259c73cc4a950a2e84141097310.svg
Ruizhe Wang
Mr-Philo
user
Mr-Philo
6799b367d30dc065a2d51593
Yeyun Gong
6799b367d30dc065a2d51594
Xiao Liu
claimed_verified
2025-01-29T17:16:05.045Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
lx865712528
user
lx865712528
6799b367d30dc065a2d51595
Guoshuai Zhao
admin_assigned
2025-01-29T17:20:36.800Z
663de80ca920d195191807da
/avatars/2437ce3fa073a07b971d370c26c7ab65.svg
Guoshuai Zhao
crayonshine
user
crayonshine
6799b367d30dc065a2d51596
Ziyue Yang
admin_assigned
2025-01-29T17:20:42.213Z
62f6a9add3bdacb7eec0d4f5
https://cdn-avatars.huggingface.co/v1/production/uploads/1660332390183-noauth.jpeg
Ziyue Yang
ziyueyang37
user
ziyueyang37
6799b367d30dc065a2d51597
Baining Guo
6799b367d30dc065a2d51598
Zhengjun Zha
6799b367d30dc065a2d51599
Peng Cheng
extracted_pending
2025-01-29T04:49:44.372Z
653feb7ccf1f9c88f4928910
/avatars/23a6a6818116683ea9485e1470a0062f.svg
Peng Cheng
cp5555
user
cp5555
Optimizing Large Language Model Training Using FP4 Quantization
The growing computational demands of training large language models (LLMs)
necessitate more efficient methods. Quantized training presents a promising
solution by enabling low-bit arithmetic operations to reduce these costs. While
FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge
due to significant quantization errors and limited representational capacity.
This work introduces the first FP4 training framework for LLMs, addressing
these challenges with two key innovations: a differentiable quantization
estimator for precise weight updates and an outlier clamping and compensation
strategy to prevent activation collapse. To ensure stability, the framework
integrates a mixed-precision training scheme and vector-wise quantization.
Experimental results demonstrate that our FP4 framework achieves accuracy
comparable to BF16 and FP8, with minimal degradation, scaling effectively to
13B-parameter LLMs trained on up to 100B tokens. With the emergence of
next-generation hardware supporting FP4, our framework sets a foundation for
efficient ultra-low precision training.
The FP4 training framework for LLMs includes differentiable quantization and outlier clamping to achieve accuracy comparable to BF16 and FP8 with minimal degradation on large-scale models.
quantized training FP4 LLMs FP8 differentiable quantization outlier clamping activation collapse mixed-precision training vector-wise quantization BF16 parameter-efficient token-wise training