So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
text
So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
new_line
new_line
It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
text
It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
new_line
new_line
However what the quantization then does with that information is where I was wrong.
text
However what the quantization then does with that information is where I was wrong.
new_line
new_line
I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
text
I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
new_line
new_line
Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
text
Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
new_line
new_line
The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
text
The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
new_line
new_line
But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
text
But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
new_line
new_line
Huge shoutout to 
text
Huge shoutout to 
@compilade
mention
compilade
 for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
text
 for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp

It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important

However what the quantization then does with that information is where I was wrong.

I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW

Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations

The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group

But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix

Huge shoutout to @compilade for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
❤️
MaziyarPanahi victor Themoneybuddha John6666 bullerwins Rybens Anderson452 NamedUser42069 qnixsynapse Clausss exdysa AtAndDev IAmTheCollector wsuff rreed-pha djuna MarinaraSpaghetti Pomni RakshitAralimatti osanseviero venketh Arakinas wonderboy nicoboss Joseph717171 AIGUYCONTENT Handgun1773 bradeenl EloyOn
👍
Aurelien-Morgan Clausss AtAndDev sroecker iandol rreed-pha ajibawa-2023 ijohn07 thiera1 nbeerbower newsletter venketh mpk63 wonderboy clem nicoboss jsfs11 Joseph717171 waldie fendiprime
🚀
rreed-pha wonderboy nicoboss Joseph717171
5fd5e18a90b6dc4633f6d292
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd5e18a90b6dc4633f6d292/gZXHW5dd9R86AV9LMZ--y.png
Maziyar Panahi
MaziyarPanahi
user
66bfd0deef1f45cd717f88fa
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_O-ycBVEVTAvhqC9wH4v2.png
Eric Sims
Themoneybuddha
user
631680bf3edbcfbc5d490227
/avatars/c6c710651ec2bde1680497afd4649281.svg
John Freier
mpk63
user
623aada982038110d90921f3
https://cdn-avatars.huggingface.co/v1/production/uploads/623aada982038110d90921f3/nSHSXJz7PYyw9fn8qoAn4.jpeg
zero
wonderboy
user
63c1dfd4a0ffa3857eb362a9
/avatars/ea4398745974d781ae9dc0e95b12cabe.svg
Joseph
Joseph717171
user