RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
Post training quantization is essential for deploying large language models (LLMs) on resource constrained hardware, yet state of the art methods enforce uniform bit widths across layers, yielding suboptimal accuracy efficiency trade offs. We present RAMP (Reinforcement Adaptive Mixed Precision), an off policy Soft Actor Critic framework that learns per layer bit width assignments to minimize perplexity under a global bit budget. The policy conditions on an 11 dimensional embedding of activation statistics, weight properties, and structural descriptors, enabling zero shot transfer across model families and scales. To enable stable sub 4 bit quantization, we introduce Scale Folding, a preconditioning technique that migrates activation outliers into weights via per channel scaling and normalization layer compensation. A quality prioritized reward with asymmetric penalties and budget cliffs drives rapid convergence. On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality. Critically, a policy trained only on Llama 2 7B generalizes zero shot to Llama 2 13B and Mistral 7B, often surpassing target specific training, supporting the hypothesis that quantization sensitivity is primarily architectural. The HALO pipeline exports allocations to GGUF format for kernel free inference on CPUs, GPUs, and edge devices, retaining 99.5% of FP16 commonsense reasoning performance.
69bb7e1aba2b62a5fb0b0d2d
Arpit Singh Gautam
claimed_verified
2026-04-08T08:06:52.942Z
64a2c460bf96ccb679a203a8
https://cdn-avatars.huggingface.co/v1/production/uploads/64a2c460bf96ccb679a203a8/PSg_6bTIg1melGbIyW5EB.jpeg
Arpit Singh Gautam
ArpitSinghGautam
user
ArpitSinghGautam
69bb7e1aba2b62a5fb0b0d2e
Saurabh Jha
claimed_verified
2026-03-23T10:56:21.562Z
66084151af425ca71e89327f
/avatars/87f435c7e49e883a23acb979f7307b81.svg
saurabh Jha
Saurabhjha21
user
Saurabhjha21
RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
Post training quantization is essential for deploying large language models (LLMs) on resource constrained hardware, yet state of the art methods enforce uniform bit widths across layers, yielding suboptimal accuracy efficiency trade offs. We present RAMP (Reinforcement Adaptive Mixed Precision), an off policy Soft Actor Critic framework that learns per layer bit width assignments to minimize perplexity under a global bit budget. The policy conditions on an 11 dimensional embedding of activation statistics, weight properties, and structural descriptors, enabling zero shot transfer across model families and scales. To enable stable sub 4 bit quantization, we introduce Scale Folding, a preconditioning technique that migrates activation outliers into weights via per channel scaling and normalization layer compensation. A quality prioritized reward with asymmetric penalties and budget cliffs drives rapid convergence. On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality. Critically, a policy trained only on Llama 2 7B generalizes zero shot to Llama 2 13B and Mistral 7B, often surpassing target specific training, supporting the hypothesis that quantization sensitivity is primarily architectural. The HALO pipeline exports allocations to GGUF format for kernel free inference on CPUs, GPUs, and edge devices, retaining 99.5% of FP16 commonsense reasoning performance.
Reinforcement learning-based mixed precision quantization method achieves superior compression efficiency and model performance for large language models through adaptive bit width assignment and novel scale folding techniques.
post training quantization large language models uniform bit widths reinforcement learning Soft Actor Critic mixed precision perplexity bit budget activation statistics weight properties structural descriptors zero shot transfer sub 4 bit quantization Scale Folding preconditioning technique asymmetric penalties budget cliffs convergence Llama 2 Mistral GGUF format kernel free inference FP16