SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.
6936a2fc3962c926cf68088c
Ruixuan Huang
6936a2fc3962c926cf68088d
Hao Zeng
6936a2fc3962c926cf68088e
Hantao Huang
6936a2fc3962c926cf68088f
Jinyuan Shi
6936a2fc3962c926cf680890
Minghui Yu
6936a2fc3962c926cf680891
Ian En-Hsu Yen
6936a2fc3962c926cf680892
Shuai Wang
SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs
Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.
A Sparse-Quantized Format (SQ-format) is proposed to improve the balance between accuracy and efficiency in post-training quantization of large language models by leveraging sparse and low-precision matrix multiplications.
Post-training quantization large language models low-bit quantization sparsification W4A8 W8A8 2:4 semi-structure sparse sparse matrix low-precision matrix multiplication SQ-format Pareto improvement performance throughput activations outlier inequality status static compression AI accelerators