OpenELM in Core ML
text
OpenELM in Core ML
new_line
new_line
Apple recently released a set of efficient LLMs in sizes varying between 270M and 3B parameters. Their quality, according to benchmarks, is similar to OLMo models of comparable size, but they required half the pre-training tokens because they use layer-wise scaling, where the number of attention heads increases in deeper layers.
text
Apple recently released a set of efficient LLMs in sizes varying between 270M and 3B parameters. Their quality, according to benchmarks, is similar to OLMo models of comparable size, but they required half the pre-training tokens because they use layer-wise scaling, where the number of attention heads increases in deeper layers.
new_line
new_line
I converted these models to Core ML, for use on Apple Silicon, using this script: 
text
I converted these models to Core ML, for use on Apple Silicon, using this script: 
https://gist.github.com/pcuenca/23cd08443460bc90854e2a6f0f575084
https://gist.github.com/pcuenca/23cd08443460bc90854e2a6f0f575084
link
. The converted models were uploaded to this community in the Hub for anyone that wants to integrate inside their apps: 
text
. The converted models were uploaded to this community in the Hub for anyone that wants to integrate inside their apps: 
https://huggingface.co/collections/corenet-community/openelm-core-ml-6630c6b19268a5d878cfd194
corenet-community/openelm-core-ml-6630c6b19268a5d878cfd194
collection
resource
https://huggingface.co/collections/corenet-community/openelm-core-ml-6630c6b19268a5d878cfd194
new_line
new_line
The conversion was done with the following parameters:
text
The conversion was done with the following parameters:
new_line
- Precision: float32.
text
- Precision: float32.
new_line
- Sequence length: fixed to 128.
text
- Sequence length: fixed to 128.
new_line
new_line
With swift-transformers (
text
With swift-transformers (
https://github.com/huggingface/swift-transformers
https://github.com/huggingface/swift-transformers
link
), I'm getting about 56 tok/s with the 270M on my M1 Max, and 6.5 with the largest 3B model. These speeds could be improved by converting to 
text
), I'm getting about 56 tok/s with the 270M on my M1 Max, and 6.5 with the largest 3B model. These speeds could be improved by converting to 
float16
`float16`
inline_code
. However, there's some precision loss somewhere and generation doesn't work in 
text
. However, there's some precision loss somewhere and generation doesn't work in 
float16
`float16`
inline_code
 mode yet. I'm looking into this and will keep you posted! Or take a look at this issue if you'd like to help: 
text
 mode yet. I'm looking into this and will keep you posted! Or take a look at this issue if you'd like to help: 
https://github.com/huggingface/swift-transformers/issues/95
https://github.com/huggingface/swift-transformers/issues/95
link
new_line
new_line
I'm also looking at optimizing inference using an experimental kv cache in swift-transformers. It's a bit tricky because the layers have varying number of attention heads, but I'm curious to see how much this feature can accelerate performance in this model family :)
text
I'm also looking at optimizing inference using an experimental kv cache in swift-transformers. It's a bit tricky because the layers have varying number of attention heads, but I'm curious to see how much this feature can accelerate performance in this model family :)
new_line
new_line
Regarding the instruct fine-tuned models, I don't know the chat template that was used. The models use the Llama 2 tokenizer, but the Llama 2 chat template, or the default Alignment Handbook one that was used to train, are not recognized. Any ideas on this welcome!
text
Regarding the instruct fine-tuned models, I don't know the chat template that was used. The models use the Llama 2 tokenizer, but the Llama 2 chat template, or the default Alignment Handbook one that was used to train, are not recognized. Any ideas on this welcome!
OpenELM in Core ML

Apple recently released a set of efficient LLMs in sizes varying between 270M and 3B parameters. Their quality, according to benchmarks, is similar to OLMo models of comparable size, but they required half the pre-training tokens because they use layer-wise scaling, where the number of attention heads increases in deeper layers.

I converted these models to Core ML, for use on Apple Silicon, using this script: https://gist.github.com/pcuenca/23cd08443460bc90854e2a6f0f575084. The converted models were uploaded to this community in the Hub for anyone that wants to integrate inside their apps: https://huggingface.co/collections/corenet-community/openelm-core-ml-6630c6b19268a5d878cfd194

The conversion was done with the following parameters:
- Precision: float32.
- Sequence length: fixed to 128.

With swift-transformers (https://github.com/huggingface/swift-transformers), I'm getting about 56 tok/s with the 270M on my M1 Max, and 6.5 with the largest 3B model. These speeds could be improved by converting to `float16`. However, there's some precision loss somewhere and generation doesn't work in `float16` mode yet. I'm looking into this and will keep you posted! Or take a look at this issue if you'd like to help: https://github.com/huggingface/swift-transformers/issues/95

I'm also looking at optimizing inference using an experimental kv cache in swift-transformers. It's a bit tricky because the layers have varying number of attention heads, but I'm curious to see how much this feature can accelerate performance in this model family :)

Regarding the instruct fine-tuned models, I don't know the chat template that was used. The models use the Llama 2 tokenizer, but the Llama 2 chat template, or the default Alignment Handbook one that was used to train, are not recognized. Any ideas on this welcome!
🔥
ZachNagengast brightbit reach-vb victor radames JimRWallace NickyNicky 1duo HelloCephalopod not-lain julien-c MathisDevFP maxrubin629 michellenicole Rainierraoul anthonymikinka toddmath irotem98 velyan qihqi jorgemunozl
❤️
reach-vb JimRWallace SuperL3D HelloCephalopod not-lain azhka MathisDevFP toddmath velyan
🚀
reach-vb HelloCephalopod not-lain MathisDevFP Norod78 0xjorgev
🤯
reach-vb HelloCephalopod not-lain MathisDevFP
63a13e4445edac9f750b7cb9
/avatars/2523b978663d915705cb9ac4cb878d6f.svg
Stephen
smpanaro
user
603d25b75f9d390ab190b777
https://cdn-avatars.huggingface.co/v1/production/uploads/1617264212503-603d25b75f9d390ab190b777.jpeg
Pedro Cuenca
pcuenq
user
639fe05cbeb95d698de716bc
/avatars/a517b6e94c8442cfa212ec0df42c9b1a.svg
Cebro
crbo
user
683861adfdfd0cef3ef9fd7f
https://cdn-avatars.huggingface.co/v1/production/uploads/683861adfdfd0cef3ef9fd7f/d6gmkoEX-BdOtny63-Yyu.jpeg
Irasubiza Viateur
VIATEUR-AI
user