RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs
Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance. Residual binarization enables hardware-friendly, matmul-free inference by stacking binary (pm1) layers, but is plagued by pathological feature co-adaptation. We identify a key failure mode, which we term inter-path adaptation: during quantization-aware training (QAT), parallel residual binary paths learn redundant features, degrading the error-compensation structure and limiting the expressive capacity of the model. While prior work relies on heuristic workarounds (e.g., path freezing) that constrain the solution space, we propose RaBiT, a novel quantization framework that resolves co-adaptation by algorithmically enforcing a residual hierarchy. Its core mechanism sequentially derives each binary path from a single shared full-precision weight, which ensures that every path corrects the error of the preceding one. This process is stabilized by a robust initialization that prioritizes functional preservation over mere weight approximation. RaBiT redefines the 2-bit accuracy-efficiency frontier: it achieves state-of-the-art performance, rivals even hardware-intensive Vector Quantization (VQ) methods, and delivers a 4.49times inference speed-up over full-precision models on an RTX 4090.
6985919b4ad556f294b7ec9c
Youngcheon You
6985919b4ad556f294b7ec9d
Banseok Lee
6985919b4ad556f294b7ec9e
Minseop Choi
6985919b4ad556f294b7ec9f
Seonyoung Kim
6985919b4ad556f294b7eca0
Hyochan Chong
claimed_verified
2026-02-06T18:50:42.521Z
6670d2ec92412fd464eac919
/avatars/f76013e72d19b12feddd80f3a4b5d71f.svg
Hyochan Chong
d7chong
user
d7chong
6985919b4ad556f294b7eca1
Changdong Kim
6985919b4ad556f294b7eca2
Youngmin Kim
6985919b4ad556f294b7eca3
Dongkyu Kim
claimed_verified
2026-02-09T08:35:27.743Z
67f25ca8cef233be93bec839
/avatars/9e0bf25d4d7b1ab2a0a59b1bf04b0d80.svg
Dongkyu Kim
dongkyu-kim
user
dongkyu-kim
RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs
Efficient deployment of large language models (LLMs) requires extreme quantization, forcing a critical trade-off between low-bit efficiency and performance. Residual binarization enables hardware-friendly, matmul-free inference by stacking binary (pm1) layers, but is plagued by pathological feature co-adaptation. We identify a key failure mode, which we term inter-path adaptation: during quantization-aware training (QAT), parallel residual binary paths learn redundant features, degrading the error-compensation structure and limiting the expressive capacity of the model. While prior work relies on heuristic workarounds (e.g., path freezing) that constrain the solution space, we propose RaBiT, a novel quantization framework that resolves co-adaptation by algorithmically enforcing a residual hierarchy. Its core mechanism sequentially derives each binary path from a single shared full-precision weight, which ensures that every path corrects the error of the preceding one. This process is stabilized by a robust initialization that prioritizes functional preservation over mere weight approximation. RaBiT redefines the 2-bit accuracy-efficiency frontier: it achieves state-of-the-art performance, rivals even hardware-intensive Vector Quantization (VQ) methods, and delivers a 4.49times inference speed-up over full-precision models on an RTX 4090.
Residual binarization framework RaBiT addresses feature co-adaptation in quantized LLMs through hierarchical path derivation and robust initialization, achieving superior accuracy-efficiency trade-offs.
residual binarization quantization-aware training QAT inter-path adaptation residual hierarchy binary layers error-compensation structure RaBiT Vector Quantization VQ inference speed-up RTX 4090