UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
Deploying large language model (LLM) models on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availability may be an issue as it is directly impacted by the current device workload, adding to the uncertainty of model deployment. We introduce UniQL, a unified post-training quantization and low-rank compression framework with on-device configurable pruning rates for edge LLMs. UniQL is a general framework that integrates quantization and low-rank compression for Transformers, State Space Models (SSMs), and hybrid models to support diverse edge applications. In our proposed joint framework, we introduce an efficient structured weight-sorting method that speeds up computation by 20x, quantization-aware singular value decomposition (SVD) to minimize quantization errors, state-aware weight sorting for SSMs, and a fused rotary positional embedding (RoPE) kernel for pruned models. Our framework performs weight-sorting, fine-tuning, and quantization in the cloud in a single-pass workflow, while enabling on-device configurable pruning rates up to 35%. Our experiments show that quantized and pruned models achieve a memory reduction of 4x-5.7x and a token-throughput improvement of 2.7x-3.4x, maintaining accuracy within 5% of the original models at 15% pruning across Transformers (Llama3 and Qwen2.5), SSMs (Mamba2), and hybrid models (Nemotron-H and Bamba-v2). The code and quantized models are available at: https://github.com/enyac-group/UniQL.
69311f792d1e5b0a7d84da37
Hung-Yueh Chiang
claimed_verified
2025-12-04T08:43:34.992Z
65ca7d8d5cf913133d3b493a
https://cdn-avatars.huggingface.co/v1/production/uploads/65ca7d8d5cf913133d3b493a/i9qx6xdYeGSuwsxgQCb7N.png
Hung-Yueh Chiang
hychiang
user
hychiang
69311f792d1e5b0a7d84da38
Chi-Chih Chang
admin_assigned
2025-12-04T11:13:39.196Z
6589f238d1331d552b1b26f5
/avatars/4a496d5791a79df9718e4a71845ae8eb.svg
Chi-Chih Chang
shadowpa0327
user
shadowpa0327
69311f792d1e5b0a7d84da39
Yu-Chen Lu
69311f792d1e5b0a7d84da3a
Chien-Yu Lin
69311f792d1e5b0a7d84da3b
Kai-Chiang Wu
69311f792d1e5b0a7d84da3c
Mohamed S. Abdelfattah
69311f792d1e5b0a7d84da3d
Diana Marculescu
UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMs
Deploying large language model (LLM) models on mobile platforms faces significant challenges due to the limited memory and shared computational resources of the device. Resource availability may be an issue as it is directly impacted by the current device workload, adding to the uncertainty of model deployment. We introduce UniQL, a unified post-training quantization and low-rank compression framework with on-device configurable pruning rates for edge LLMs. UniQL is a general framework that integrates quantization and low-rank compression for Transformers, State Space Models (SSMs), and hybrid models to support diverse edge applications. In our proposed joint framework, we introduce an efficient structured weight-sorting method that speeds up computation by 20x, quantization-aware singular value decomposition (SVD) to minimize quantization errors, state-aware weight sorting for SSMs, and a fused rotary positional embedding (RoPE) kernel for pruned models. Our framework performs weight-sorting, fine-tuning, and quantization in the cloud in a single-pass workflow, while enabling on-device configurable pruning rates up to 35%. Our experiments show that quantized and pruned models achieve a memory reduction of 4x-5.7x and a token-throughput improvement of 2.7x-3.4x, maintaining accuracy within 5% of the original models at 15% pruning across Transformers (Llama3 and Qwen2.5), SSMs (Mamba2), and hybrid models (Nemotron-H and Bamba-v2). The code and quantized models are available at: https://github.com/enyac-group/UniQL.
UniQL, a unified post-training quantization and low-rank compression framework, enhances the deployment of large language models on mobile devices by reducing memory usage and improving token throughput while maintaining accuracy.
post-training quantization low-rank compression edge LLMs Transformers State Space Models hybrid models structured weight-sorting quantization-aware singular value decomposition state-aware weight sorting fused rotary positional embedding on-device configurable pruning rates