Transformers documentation
SINQ
Get started
Base classes
Models
Preprocessors
Inference
Pipeline API
Generate API
Optimization
Chat with models
Serving
Training
Hardware
Quantization
OverviewQuantization conceptsChoose a method OptimumContribute a quantization method
Common tasks
Quantize while loading
bitsandbytesEETQFBGEMM FP8Fine-grained FP8Four Over SixFP-QuantGPTQHIGGSHQQMetalMXFP4NVFP4QuantoSINQtorchao
Load pre-quantized checkpoints
Ecosystem integrations
Resources
API
SINQ
Sinkhorn-Normalized Quantization (SINQ) balances the scale of each weight matrix’s rows and columns before quantizing it, which reduces the error from outliers without calibration data. Use it as a calibration-free alternative to HQQ for 2- to 8-bit weights on a single GPU.
The table below summarizes what SINQ supports in Transformers.
| Config | SinqConfig |
quant_method | sinq |
| Workflow | Quantize while loading |
| Bits | 2, 3, 4, 5, 6, 8 |
| Hardware | One CUDA GPU, CPU |
| Save | ✅ |
| Fine-tune | ✅ |
| Auto-compile | ❌ |
Install SINQ
Install the sinq package.
pip install sinq
Quantize a model with SINQ
Create a SinqConfig and pass it to from_pretrained(). The example below quantizes a model to 4 bits and keeps lm_head in bf16.
from transformers import AutoModelForCausalLM, SinqConfig
quantization_config = SinqConfig(nbits=4, group_size=64, modules_to_not_convert=["lm_head"])
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-1.7B",
quantization_config=quantization_config,
device_map="cuda:0",
)Without a CUDA GPU, SINQ quantizes and runs the model on the CPU, which is much slower.
Configure SINQ
These SinqConfig parameters change how the weights are quantized.
| Parameter | Default | Effect |
|---|---|---|
nbits | 4 | Number of bits per weight. |
group_size | 64 | Number of weights that share a scale. Use a multiple of 8. |
tiling_mode | "1D" | How the weight matrix is split into tiles, either "1D" or "2D". |
modules_to_not_convert | None | Modules to leave unquantized. With the default, every linear layer is quantized, including lm_head. |
Limitations
- SINQ runs on one device. A
device_mapthat spreads the model across several devices, including CPU offloading, raises aRuntimeError. - Transformers supports only the calibration-free
method="sinq". Quantizing with the calibratedmethod="asinq"raises aValueError. Use the SINQ library to quantize with A-SINQ.
Resources
- SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights paper
- SINQ repository