Qwen3.5 Performance Metrics
text
Qwen3.5 Performance Metrics
new_line
new_line
With the 3.5 architecture, a lot of the old quanting methods don't work as before. I noticed this when benchmarking Deckard(qx) quants and by mistake ran a q8 that was better. That only happens if the qx sucked--and it did--enhancing layers just because they look interesting doesn't work anymore, so until I get a clear understanding of the architecture, I will publish mxfp4 and mxfp8 of the 3.5 models, that seem very stable and high performant
text
With the 3.5 architecture, a lot of the old quanting methods don't work as before. I noticed this when benchmarking Deckard(qx) quants and by mistake ran a q8 that was better. That only happens if the qx sucked--and it did--enhancing layers just because they look interesting doesn't work anymore, so until I get a clear understanding of the architecture, I will publish mxfp4 and mxfp8 of the 3.5 models, that seem very stable and high performant
new_line
new_line
I will start posting here the metrics I gather from the series, starting with the smallest. If I have numbers from previous or similar models, I will post them in comparison
text
I will start posting here the metrics I gather from the series, starting with the smallest. If I have numbers from previous or similar models, I will post them in comparison
new_line
new_line
Qwen3.5-0.8B
text
Qwen3.5-0.8B
new_line
new_line
quant    arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.351,0.501,0.733,0.462,0.348,0.682,0.573
mxfp4    0.339,0.489,0.738,0.433,0.330,0.672,0.553

Old model performance

Qwen3-0.6B
bf16     0.298,0.354,0.378,0.415,0.344,0.649,0.534
q8-hi    0.296,0.355,0.378,0.416,0.348,0.652,0.529
q8       0.299,0.354,0.378,0.414,0.346,0.650,0.535
q6-hi    0.301,0.356,0.378,0.415,0.350,0.651,0.541
q6       0.300,0.367,0.378,0.416,0.344,0.647,0.524
mxfp4    0.286,0.364,0.609,0.404,0.316,0.626,0.531

Quant    Perplexity     Peak memory
mxfp8    6.611 ± 0.049  7.65 GB
mxfp4    7.455 ± 0.057  6.33 GB
brainwaves
```brainwaves
quant    arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.351,0.501,0.733,0.462,0.348,0.682,0.573
mxfp4    0.339,0.489,0.738,0.433,0.330,0.672,0.553

Old model performance

Qwen3-0.6B
bf16     0.298,0.354,0.378,0.415,0.344,0.649,0.534
q8-hi    0.296,0.355,0.378,0.416,0.348,0.652,0.529
q8       0.299,0.354,0.378,0.414,0.346,0.650,0.535
q6-hi    0.301,0.356,0.378,0.415,0.350,0.651,0.541
q6       0.300,0.367,0.378,0.416,0.344,0.647,0.524
mxfp4    0.286,0.364,0.609,0.404,0.316,0.626,0.531

Quant    Perplexity     Peak memory
mxfp8    6.611 ± 0.049  7.65 GB
mxfp4    7.455 ± 0.057  6.33 GB
```
code_fence
new_line
new_line
Detailed metrics by model
text
Detailed metrics by model
new_line
new_line
https://huggingface.co/nightmedia/Qwen3.5-0.8B-mxfp8-mlx
nightmedia/Qwen3.5-0.8B-mxfp8-mlx
model
resource
https://huggingface.co/nightmedia/Qwen3.5-0.8B-mxfp8-mlx
new_line
new_line
https://huggingface.co/nightmedia/Qwen3.5-2B-mxfp8-mlx
nightmedia/Qwen3.5-2B-mxfp8-mlx
model
resource
https://huggingface.co/nightmedia/Qwen3.5-2B-mxfp8-mlx
new_line
new_line
https://huggingface.co/nightmedia/Qwen3.5-4B-mxfp8-mlx
nightmedia/Qwen3.5-4B-mxfp8-mlx
model
resource
https://huggingface.co/nightmedia/Qwen3.5-4B-mxfp8-mlx
new_line
new_line
https://huggingface.co/nightmedia/Qwen3.5-9B-mxfp8-mlx
nightmedia/Qwen3.5-9B-mxfp8-mlx
model
resource
https://huggingface.co/nightmedia/Qwen3.5-9B-mxfp8-mlx
new_line
new_line
https://huggingface.co/nightmedia/Qwen3.5-27B-Text
nightmedia/Qwen3.5-27B-Text
model
resource
https://huggingface.co/nightmedia/Qwen3.5-27B-Text
new_line
new_line
https://huggingface.co/nightmedia/Qwen3.5-122B-A10B-Text-mxfp4-mlx
nightmedia/Qwen3.5-122B-A10B-Text-mxfp4-mlx
model
resource
https://huggingface.co/nightmedia/Qwen3.5-122B-A10B-Text-mxfp4-mlx
new_line
new_line
More metrics coming soon.
text
More metrics coming soon.
new_line
new_line
I am running these on my Mac, an M4Max with 128GB RAM. Some performance numbers like tokens/second reflect the performance on my box.
text
I am running these on my Mac, an M4Max with 128GB RAM. Some performance numbers like tokens/second reflect the performance on my box.
new_line
new_line
This post will be updated with every model that gets tested. The larger models take hours, the 27B a couple days, so it will be a long process.
text
This post will be updated with every model that gets tested. The larger models take hours, the 27B a couple days, so it will be a long process.
new_line
new_line
-G
text
-G
Qwen3.5 Performance Metrics

With the 3.5 architecture, a lot of the old quanting methods don't work as before. I noticed this when benchmarking Deckard(qx) quants and by mistake ran a q8 that was better. That only happens if the qx sucked--and it did--enhancing layers just because they look interesting doesn't work anymore, so until I get a clear understanding of the architecture, I will publish mxfp4 and mxfp8 of the 3.5 models, that seem very stable and high performant

I will start posting here the metrics I gather from the series, starting with the smallest. If I have numbers from previous or similar models, I will post them in comparison

Qwen3.5-0.8B

```brainwaves
quant    arc   arc/e boolq hswag obkqa piqa  wino
mxfp8    0.351,0.501,0.733,0.462,0.348,0.682,0.573
mxfp4    0.339,0.489,0.738,0.433,0.330,0.672,0.553

Old model performance

Qwen3-0.6B
bf16     0.298,0.354,0.378,0.415,0.344,0.649,0.534
q8-hi    0.296,0.355,0.378,0.416,0.348,0.652,0.529
q8       0.299,0.354,0.378,0.414,0.346,0.650,0.535
q6-hi    0.301,0.356,0.378,0.415,0.350,0.651,0.541
q6       0.300,0.367,0.378,0.416,0.344,0.647,0.524
mxfp4    0.286,0.364,0.609,0.404,0.316,0.626,0.531

Quant    Perplexity     Peak memory
mxfp8    6.611 ± 0.049  7.65 GB
mxfp4    7.455 ± 0.057  6.33 GB
```

Detailed metrics by model

https://huggingface.co/nightmedia/Qwen3.5-0.8B-mxfp8-mlx

https://huggingface.co/nightmedia/Qwen3.5-2B-mxfp8-mlx

https://huggingface.co/nightmedia/Qwen3.5-4B-mxfp8-mlx

https://huggingface.co/nightmedia/Qwen3.5-9B-mxfp8-mlx

https://huggingface.co/nightmedia/Qwen3.5-27B-Text

https://huggingface.co/nightmedia/Qwen3.5-122B-A10B-Text-mxfp4-mlx

More metrics coming soon.

I am running these on my Mac, an M4Max with 128GB RAM. Some performance numbers like tokens/second reflect the performance on my box.

This post will be updated with every model that gets tested. The larger models take hours, the 27B a couple days, so it will be a long process.

-G
🚀
John6666
67b0caceb06805a4370c44c5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/AiTL7ds0r5S7A-qgsON5y.jpeg
Gheorghe Chesler
nightmedia
user