Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
text
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)
new_line
new_line
The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
text
The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16
new_line
new_line
Had claude make me a script, using the new Reflection-70B, and these are the results:
text
Had claude make me a script, using the new Reflection-70B, and these are the results:
new_line
new_line
Total weights: 70553706496
text
Total weights: 70553706496
new_line
Fully representable: 70530215524
text
Fully representable: 70530215524
new_line
Squashed: 23490972
text
Squashed: 23490972
new_line
Percentage squashed: 0.03%
text
Percentage squashed: 0.03%
new_line
new_line
0.03%!!!!
text
0.03%!!!!
new_line
new_line
A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
text
A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)
new_line
new_line
This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
text
This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.
new_line
new_line
Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
text
Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
Decided to try to check how many weights in a 70b F32 model would be squashed when converted to F16 (spoiler, it's shockingly few)

The reason for this comparison is that it should represent the same percentage of squishing as bf16 to fp16

Had claude make me a script, using the new Reflection-70B, and these are the results:

Total weights: 70553706496
Fully representable: 70530215524
Squashed: 23490972
Percentage squashed: 0.03%

0.03%!!!!

A couple things to note, this uses a roundtrip of F32 -> F16 -> F32 and then torch.isclose to account for rounding errors that come up by the very nature of extremely accurate numbers, but it uses VERY small tolerances (rtol=1e-5, atol=1e-8)

This is also examining EVERY weight that was stored at F32, and for most layers I was somewhere between 0% and 0.03% of weights being squashed, no major outliers.

Overall, I feel even safer converting to F16 for llama.cpp, the extremely small number of weights that fall outside the range are likely so small that they don't actually play a role in the final output of the model at inference anyways.
🔥
prithivMLmods Sri-Vigneshwar-DJ Joseph717171 not-lain John6666 morph3v5
❤️
Joseph717171 KhaldiAbderrhmane nicoboss
👍
MoonRide blotfaba nicoboss
👀
John6666 Joseph717171
🤯
louisbrulenaudet Joseph717171
🤗
Joseph717171
🧠
Joseph717171
🚀
Joseph717171
64a9896531a43d726502b9f5
/avatars/da52e5fce67042332fa1e9f5fd3e5635.svg
Luke Chadwick
vertis
user
6435718aaaef013d1aec3b8b
https://cdn-avatars.huggingface.co/v1/production/uploads/6435718aaaef013d1aec3b8b/XKf-8MA47tjVAM6SCX0MP.jpeg
Bartowski
bartowski
user
6640bbd0220cfa8cbfdce080
https://cdn-avatars.huggingface.co/v1/production/uploads/6640bbd0220cfa8cbfdce080/wiAHUu5ewawyipNs0YFBR.png
John Smith
John6666
user
63c1dfd4a0ffa3857eb362a9
/avatars/ea4398745974d781ae9dc0e95b12cabe.svg
Joseph
Joseph717171
user
666b14d48ccf7f0aa469f3bd
/avatars/99351620d65d263418e6d0d4e170f055.svg
Abrosimov
ajiriro
user
66d1efa935c36f266f507cff
https://cdn-avatars.huggingface.co/v1/production/uploads/66d1efa935c36f266f507cff/a2-fPLeGwAp5fqCKdqfzp.jpeg
Harmendo
Hampetiudo
user
63ef9ccae2319f35318ca8de
/avatars/3b03217c22442b7bfed9beac2bf50d17.svg
Alex Daminger
Handgun1773
user
6570493c84632c95659d342a
/avatars/98d7cbc7bf4cbf4f2810cbc0a1a34d64.svg
Iwan Kawrakow
ikawrakow
user
655e9450d5c0d3db536859d1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/4Az8a8F60rNOD3L3ThsCe.png
Compilade
compilade
user