INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization
  Formats
Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly
embracing low-precision floating-point (FP) formats to handle the pervasive
activation outliers in Large Language Models (LLMs). Despite this industry
trend, a unified comparison of FP and integer (INT) quantization across varying
granularities has been missing, leaving algorithm and hardware co-design
without clear guidance. This paper fills that gap by systematically
investigating the trade-offs between FP and INT formats. We reveal a critical
performance crossover: while FP excels in coarse-grained quantization, the
comparison at fine-grained (block-wise) levels is more nuanced. Our
comprehensive comparison demonstrates that for popular 8-bit fine-grained
formats (e.g., MX with block size 32), MXINT8 is superior to its FP counterpart
in both algorithmic accuracy and hardware efficiency. However, for 4-bit
formats, FP (e.g., MXFP4, NVFP4) often holds an accuracy advantage , though we
show that NVINT4 can surpass NVFP4 when outlier-mitigation techniques like
Hadamard rotation are applied. We also introduce a symmetric clipping method
that resolves gradient bias in fine-grained low-bit INT training, enabling
nearly lossless performance for MXINT8 training. These findings challenge the
current hardware trajectory, demonstrating that a one-size-fits-all FP approach
is suboptimal and advocating that fine-grained INT formats, particularly
MXINT8, offer a better balance of accuracy, power, and efficiency for future AI
accelerators.
69081d74812eca10f9cc5e7a
Mengzhao Chen
69081d74812eca10f9cc5e7b
Meng Wu
claimed_verified
2025-11-03T20:52:16.967Z
690821568b3b900d6e63c56d
/avatars/488621d29b93f4ca0a007a068dfc64be.svg
Meng Wu
MarlinWu
user
MarlinWu
69081d74812eca10f9cc5e7c
Hui Jin
69081d74812eca10f9cc5e7d
Zhihang Yuan
69081d74812eca10f9cc5e7e
Jing Liu
69081d74812eca10f9cc5e7f
Chaoyi Zhang
69081d74812eca10f9cc5e80
Yunshui Li
69081d74812eca10f9cc5e81
Jie Huang
69081d74812eca10f9cc5e82
Jin Ma
69081d74812eca10f9cc5e83
Zeyue Xue
69081d74812eca10f9cc5e84
Zhiheng Liu
69081d74812eca10f9cc5e85
Xingyan Bin
69081d74812eca10f9cc5e86
Ping Luo
INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization
  Formats
Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly
embracing low-precision floating-point (FP) formats to handle the pervasive
activation outliers in Large Language Models (LLMs). Despite this industry
trend, a unified comparison of FP and integer (INT) quantization across varying
granularities has been missing, leaving algorithm and hardware co-design
without clear guidance. This paper fills that gap by systematically
investigating the trade-offs between FP and INT formats. We reveal a critical
performance crossover: while FP excels in coarse-grained quantization, the
comparison at fine-grained (block-wise) levels is more nuanced. Our
comprehensive comparison demonstrates that for popular 8-bit fine-grained
formats (e.g., MX with block size 32), MXINT8 is superior to its FP counterpart
in both algorithmic accuracy and hardware efficiency. However, for 4-bit
formats, FP (e.g., MXFP4, NVFP4) often holds an accuracy advantage , though we
show that NVINT4 can surpass NVFP4 when outlier-mitigation techniques like
Hadamard rotation are applied. We also introduce a symmetric clipping method
that resolves gradient bias in fine-grained low-bit INT training, enabling
nearly lossless performance for MXINT8 training. These findings challenge the
current hardware trajectory, demonstrating that a one-size-fits-all FP approach
is suboptimal and advocating that fine-grained INT formats, particularly
MXINT8, offer a better balance of accuracy, power, and efficiency for future AI
accelerators.
A comprehensive comparison of low-precision floating-point and integer quantization in large language models reveals that fine-grained integer formats, especially MXINT8, offer superior accuracy and efficiency over floating-point formats in many cases.
low-precision floating-point integer quantization coarse-grained quantization fine-grained quantization block-wise quantization MXINT8 MXFP4 NVFP4 NVINT4 Hadamard rotation symmetric clipping gradient bias large language models