NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models
Weight-only quantization has become a standard approach for efficiently serving large language models (LLMs). However, existing methods fail to efficiently compress models to binary (1-bit) levels, as they either require large amounts of data and compute or incur additional storage. In this work, we propose NanoQuant, the first post-training quantization (PTQ) method to compress LLMs to both binary and sub-1-bit levels. NanoQuant formulates quantization as a low-rank binary factorization problem, and compresses full-precision weights to low-rank binary matrices and scales. Specifically, it utilizes an efficient alternating direction method of multipliers (ADMM) method to precisely initialize latent binary matrices and scales, and then tune the initialized parameters through a block and model reconstruction process. Consequently, NanoQuant establishes a new Pareto frontier in low-memory post-training quantization, achieving state-of-the-art accuracy even at sub-1-bit compression rates. NanoQuant makes large-scale deployment feasible on consumer hardware. For example, it compresses Llama2-70B by 25.8times in just 13 hours on a single H100, enabling a 70B model to operate on a consumer 8 GB GPU.
69894d2abeecc443208d2610
Hyochan Chong
claimed_verified
2026-02-09T08:29:50.005Z
6670d2ec92412fd464eac919
/avatars/f76013e72d19b12feddd80f3a4b5d71f.svg
Hyochan Chong
d7chong
user
d7chong
69894d2abeecc443208d2611
Dongkyu Kim
claimed_verified
2026-02-10T09:07:47.203Z
67f25ca8cef233be93bec839
/avatars/9e0bf25d4d7b1ab2a0a59b1bf04b0d80.svg
Dongkyu Kim
dongkyu-kim
user
dongkyu-kim
69894d2abeecc443208d2612
Changdong Kim
69894d2abeecc443208d2613
Minseop Choi
NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models
Weight-only quantization has become a standard approach for efficiently serving large language models (LLMs). However, existing methods fail to efficiently compress models to binary (1-bit) levels, as they either require large amounts of data and compute or incur additional storage. In this work, we propose NanoQuant, the first post-training quantization (PTQ) method to compress LLMs to both binary and sub-1-bit levels. NanoQuant formulates quantization as a low-rank binary factorization problem, and compresses full-precision weights to low-rank binary matrices and scales. Specifically, it utilizes an efficient alternating direction method of multipliers (ADMM) method to precisely initialize latent binary matrices and scales, and then tune the initialized parameters through a block and model reconstruction process. Consequently, NanoQuant establishes a new Pareto frontier in low-memory post-training quantization, achieving state-of-the-art accuracy even at sub-1-bit compression rates. NanoQuant makes large-scale deployment feasible on consumer hardware. For example, it compresses Llama2-70B by 25.8times in just 13 hours on a single H100, enabling a 70B model to operate on a consumer 8 GB GPU.
NanoQuant enables efficient post-training quantization of large language models to binary and sub-1-bit levels using low-rank binary factorization and ADMM optimization, achieving state-of-the-art accuracy while reducing memory requirements for consumer hardware deployment.
post-training quantization low-rank binary factorization alternating direction method of multipliers binary quantization sub-1-bit compression latent binary matrices block reconstruction model reconstruction