ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression
We present ROCKET, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. ROCKET consistently outperforms existing compression approaches across different model architectures at 20-50\% compression rates. Notably, it retains over 90\% of the original model's performance at 30\% compression without any fine-tuning. Moreover, when applying a light fine-tuning phase, recovery is substantially enhanced: for instance, compressing Qwen3-14B to an 8B-parameter model and healing it with just 30 million tokens yields performance nearly on par with the original Qwen3-8B. The code for ROCKET is at github.com/mts-ai/ROCKET/tree/main.
698d707d65c0d15a6d16224f
Ammar Ali
claimed_verified
2026-02-12T13:26:05.919Z
6166db59f78a267701a78c2a
/avatars/8784efc36f67719e9455b1f081340ed9.svg
Ammar Ali
ammarali32
user
ammarali32
698d707d65c0d15a6d162250
Baher Mohammad
698d707d65c0d15a6d162251
Denis Makhov
claimed_verified
2026-02-18T09:06:47.223Z
67bc9a9ed9a970de04255711
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_WzRhjLJnkf_ALkP4BuM_.png
Denis Makhov
dennismak1994
user
dennismak1994
698d707d65c0d15a6d162252
Dmitriy Shopkhoev
claimed_verified
2026-02-18T09:06:49.035Z
66465dfa508db0bde50d95f2
/avatars/8b4a583dc0f3cab0f1cd9a1be3daa01b.svg
Dmitry Shopkhoev
dimitriish
user
dimitriish
698d707d65c0d15a6d162253
Magauiya Zhussip
698d707d65c0d15a6d162254
Stamatios Lefkimmiatis
ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression
We present ROCKET, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. ROCKET consistently outperforms existing compression approaches across different model architectures at 20-50\% compression rates. Notably, it retains over 90\% of the original model's performance at 30\% compression without any fine-tuning. Moreover, when applying a light fine-tuning phase, recovery is substantially enhanced: for instance, compressing Qwen3-14B to an 8B-parameter model and healing it with just 30 million tokens yields performance nearly on par with the original Qwen3-8B. The code for ROCKET is at github.com/mts-ai/ROCKET/tree/main.
ROCKET is a training-free model compression method that formulates layer-wise compression as a multi-choice knapsack problem and uses sparse matrix factorization for efficient weight sparsification without iterative optimization.
model compression multi-choice knapsack problem sparse matrix factorization dictionary learning weight sparsification activation-weights sensitivity least squares closed form update fine-tuning parameter-efficient fine-tuning