As some of you know, I try to convert models to either fp32 or bf16 depending on theirs size before doing imatrix and quantization
text
As some of you know, I try to convert models to either fp32 or bf16 depending on theirs size before doing imatrix and quantization
new_line
new_line
Today I decided to see if that matters, and the results have me.. for lack of a better word, perplexed
text
Today I decided to see if that matters, and the results have me.. for lack of a better word, perplexed
new_line
new_line
My setup:
text
My setup:
new_line
new_line
Mistral Nemo Instruct 2407
text
Mistral Nemo Instruct 2407
new_line
- convert to FP32, calculate imatrix, quantize to Q8_0 and Q4_K_M
text
- convert to FP32, calculate imatrix, quantize to Q8_0 and Q4_K_M
new_line
- convert to FP16, calculate imatrix, quantize to Q8_0 and Q4_K_M
text
- convert to FP16, calculate imatrix, quantize to Q8_0 and Q4_K_M
new_line
new_line
I calculated the kld base from the FP32 model:
text
I calculated the kld base from the FP32 model:
new_line
./llama-perplexity -m /models/Mistral-Nemo-Instruct-2407-f32.gguf -f /training_data/wikitext-2-raw/wiki.test.raw --kl-divergence-base /training_data/mistral-nemo-f32.kld -ngl 35 -fa -sm row
`./llama-perplexity -m /models/Mistral-Nemo-Instruct-2407-f32.gguf -f /training_data/wikitext-2-raw/wiki.test.raw --kl-divergence-base /training_data/mistral-nemo-f32.kld -ngl 35 -fa -sm row`
inline_code
new_line
new_line
then calculated the divergence itself for each like so:
text
then calculated the divergence itself for each like so:
new_line
./llama-perplexity -m /models/Mistral-Nemo-Instruct-2407-Q8_0.gguf -f /training_data/wikitext-2-raw/wiki.test.raw --kl-divergence-base /training_data/mistral-nemo-f32.kld --kl-divergence -ngl 50 -fa -sm row
`./llama-perplexity -m /models/Mistral-Nemo-Instruct-2407-Q8_0.gguf -f /training_data/wikitext-2-raw/wiki.test.raw --kl-divergence-base /training_data/mistral-nemo-f32.kld --kl-divergence -ngl 50 -fa -sm row`
inline_code
new_line
new_line
Q4_K_M from fp16 and fp32 were similar, trading blows across statistics, odd since i expected fp32 to be strictly better but it's not
text
Q4_K_M from fp16 and fp32 were similar, trading blows across statistics, odd since i expected fp32 to be strictly better but it's not
new_line
new_line
Q8_0 is where things get weird. Despite each file being slightly different size, and the sha256sum of course being different, they each get *completely identical* scores, down to 6 decimal places of precision on the statistics.
text
Q8_0 is where things get weird. Despite each file being slightly different size, and the sha256sum of course being different, they each get *completely identical* scores, down to 6 decimal places of precision on the statistics.
new_line
new_line
How is this possible? Is there something I don't understand about llama.cpp that makes it always convert to fp16 before it does quantization? Am I wasting time using FP32/BF16??
text
How is this possible? Is there something I don't understand about llama.cpp that makes it always convert to fp16 before it does quantization? Am I wasting time using FP32/BF16??
As some of you know, I try to convert models to either fp32 or bf16 depending on theirs size before doing imatrix and quantization

Today I decided to see if that matters, and the results have me.. for lack of a better word, perplexed

My setup:

Mistral Nemo Instruct 2407
- convert to FP32, calculate imatrix, quantize to Q8_0 and Q4_K_M
- convert to FP16, calculate imatrix, quantize to Q8_0 and Q4_K_M

I calculated the kld base from the FP32 model:
`./llama-perplexity -m /models/Mistral-Nemo-Instruct-2407-f32.gguf -f /training_data/wikitext-2-raw/wiki.test.raw --kl-divergence-base /training_data/mistral-nemo-f32.kld -ngl 35 -fa -sm row`

then calculated the divergence itself for each like so:
`./llama-perplexity -m /models/Mistral-Nemo-Instruct-2407-Q8_0.gguf -f /training_data/wikitext-2-raw/wiki.test.raw --kl-divergence-base /training_data/mistral-nemo-f32.kld --kl-divergence -ngl 50 -fa -sm row`

Q4_K_M from fp16 and fp32 were similar, trading blows across statistics, odd since i expected fp32 to be strictly better but it's not

Q8_0 is where things get weird. Despite each file being slightly different size, and the sha256sum of course being different, they each get *completely identical* scores, down to 6 decimal places of precision on the statistics.

How is this possible? Is there something I don't understand about llama.cpp that makes it always convert to fp16 before it does quantization? Am I wasting time using FP32/BF16??
👍
Rybens John6666 Nelathan YaTharThShaRma999 MarinaraSpaghetti DeathGodlike osanseviero julien-c pcuenq mlabonne Joseph717171
🔥
nazimali osanseviero julien-c Joseph717171
❤️
ijohn07 Joseph717171
6435718aaaef013d1aec3b8b
https://cdn-avatars.huggingface.co/v1/production/uploads/6435718aaaef013d1aec3b8b/XKf-8MA47tjVAM6SCX0MP.jpeg
Bartowski
bartowski
user
63036c501dd5d3c62480dd14
/avatars/19801a674af5af87bf036aed6a69cce7.svg
Johannes Gäßler
JohannesGaessler
user
65ea44635b64331c067d3751
https://cdn-avatars.huggingface.co/v1/production/uploads/65ea44635b64331c067d3751/yCim-7c3tm67o5wWP_6cE.jpeg
David Belton
DavidAU
user
64aea8ff67511bd3d965697b
https://cdn-avatars.huggingface.co/v1/production/uploads/64aea8ff67511bd3d965697b/Jxn52EmDF5RApJh8antxn.jpeg
Feynman Innovations
ajibawa-2023
user
65cccccefb8ab7fcc2c6424c
https://cdn-avatars.huggingface.co/v1/production/uploads/65cccccefb8ab7fcc2c6424c/0dlk5hmzNhTWr8j9E1DXP.jpeg
Rodri Mora
bullerwins
user
65a5ad3c0b5704678a8612b9
https://cdn-avatars.huggingface.co/v1/production/uploads/65a5ad3c0b5704678a8612b9/Fh6TDWyf9EGt6U2_G-v4Q.png
AItomek ;P
altomek
user