Instructions to use Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Local Apps Settings

How to use Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ

SGLang

How to use Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ with Docker Model Runner:
```
docker model run hf.co/Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ
```

Vishva007/Qwen3.5-9B-W4A16-AutoRound-GPTQ

This is a W4A16 (4-bit weight, 16-bit activation) quantized version of Qwen/Qwen3.5-9B, produced using AutoRound — Intel's sign gradient descent based quantization method designed for production-grade accuracy retention. MTP Enabled model quantization

Quantization Details

Parameter	Value
Method	AutoRound (W4A16)
Group Size	32
Symmetric	Yes
Iterations	1200
Calibration Samples	512
Sequence Length	4096
Torch Compile	Enabled

Key Notes

High accuracy configuration — 1200 iterations with 512 calibration samples targets production-grade quality with minimal degradation from the base model.
W4A16 — Weights are quantized to 4-bit integers; activations remain in FP16 for inference stability.
~50% memory reduction compared to the FP16 base model, enabling deployment on consumer and mid-range GPUs.
Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

MTP / Speculative Decoding

This model supports Multi-Token Prediction (MTP) for improved inference throughput using speculative decoding.

When serving with compatible backends (e.g., vLLM), enable MTP using:

--speculative_config '{"method":"mtp","num_speculative_tokens":1}'

Notes

num_speculative_tokens=1 is a stable default for balancing speed and accuracy.
You can experiment with higher values for better throughput, depending on your hardware and latency requirements.

Usage

This model is compatible with transformers and backends that support AutoRound GPTQ-format weights (e.g., vLLM, SGLang, AutoGPTQ). For full model details, architecture, and capabilities, refer to the base model page.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5–$500 in free credits when you add your first $10.

PyTorch 2.14

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`	`yk0y6j6rpg`
PyTorch 2.14 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.14-runpod`	`gsp4gwx0nw`

PyTorch 2.13

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`	`y3j8xvk4f4`
PyTorch 2.13 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.13-runpod`	`vigpissn5w`

PyTorch 2.12

Template	CUDA Version	Docker Image	Template ID
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`	`qjko5yiwzi`
PyTorch 2.12 (CUDA 13.2)	13.2	`vishva123/cuda-13.2-pytorch-2.12-runpod`	`ifg6xmye0f`