Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In this work, we investigate their interaction and show that 1.58-bit BitNet is naturally more compatible with N:M sparsity than full-precision models. To study this effect, we propose Sparse-BitNet, a unified framework that jointly applies 1.58-bit quantization and dynamic N:M sparsification while ensuring stable training for the first time. Across multiple model scales and training regimes (sparse pretraining and dense-to-sparse schedules), 1.58-bit BitNet consistently exhibits smaller performance degradation than full-precision baselines at the same sparsity levels and can tolerate higher structured sparsity before accuracy collapse. Moreover, using our custom sparse tensor core, Sparse-BitNet achieves substantial speedups in both training and inference, reaching up to 1.30X. These results highlight that combining extremely low-bit quantization with semi-structured N:M sparsity is a promising direction for efficient LLMs. Code available at https://github.com/AAzdi/Sparse-BitNet
69afb99cb3ab31dee1281ed4
Di Zhang
69afb99cb3ab31dee1281ed5
Xun Wu
69afb99cb3ab31dee1281ed6
Shaohan Huang
69afb99cb3ab31dee1281ed7
Yudong Wang
69afb99cb3ab31dee1281ed8
Hanyong Shao
69afb99cb3ab31dee1281ed9
Yingbo Hao
69afb99cb3ab31dee1281eda
Zewen Chi
69afb99cb3ab31dee1281edb
Li Dong
claimed_verified
2026-03-18T08:46:29.688Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
unilm
user
unilm
69afb99cb3ab31dee1281edc
Ting Song
69afb99cb3ab31dee1281edd
Yan Xia
69afb99cb3ab31dee1281ede
Zhifang Sui
69afb99cb3ab31dee1281edf
Furu Wei
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In this work, we investigate their interaction and show that 1.58-bit BitNet is naturally more compatible with N:M sparsity than full-precision models. To study this effect, we propose Sparse-BitNet, a unified framework that jointly applies 1.58-bit quantization and dynamic N:M sparsification while ensuring stable training for the first time. Across multiple model scales and training regimes (sparse pretraining and dense-to-sparse schedules), 1.58-bit BitNet consistently exhibits smaller performance degradation than full-precision baselines at the same sparsity levels and can tolerate higher structured sparsity before accuracy collapse. Moreover, using our custom sparse tensor core, Sparse-BitNet achieves substantial speedups in both training and inference, reaching up to 1.30X. These results highlight that combining extremely low-bit quantization with semi-structured N:M sparsity is a promising direction for efficient LLMs. Code available at https://github.com/AAzdi/Sparse-BitNet
Sparse-BitNet demonstrates that 1.58-bit quantization works better with N:M sparsity than full-precision models, achieving stable training and improved efficiency across different scales and regimes.
semi-structured N:M sparsity low-bit quantization BitNet sparse tensor core dynamic N:M sparsification structured sparsity accuracy collapse sparse pretraining dense-to-sparse schedules