Recently, Essential AI released a new 8B base model 
text
Recently, Essential AI released a new 8B base model 
https://huggingface.co/EssentialAI/rnj-1
EssentialAI/rnj-1
model
resource
https://huggingface.co/EssentialAI/rnj-1
 they highlighted the importance of data mix for pretraning -
text
 they highlighted the importance of data mix for pretraning -
new_line
new_line
"In the long run, we expect our methods to automatically represent, transform, and blend data to optimize measurable abilities in pre-training. Our work on modeling data taxonomies led to new approaches for jointly clustering and mixing data distributions under data repetition penalties. Many improvements in our STEM abilities can be traced back to this. "
text
"In the long run, we expect our methods to automatically represent, transform, and blend data to optimize measurable abilities in pre-training. Our work on modeling data taxonomies led to new approaches for jointly clustering and mixing data distributions under data repetition penalties. Many improvements in our STEM abilities can be traced back to this. "
new_line
new_line
This resonates with the recent work we did around optimal dataset mixing for pretraining where we saw have the right mix can increase the efficiency of training - 
text
This resonates with the recent work we did around optimal dataset mixing for pretraining where we saw have the right mix can increase the efficiency of training - 
new_line
https://huggingface.co/blog/codelion/optimal-dataset-mixing
https://huggingface.co/blog/codelion/optimal-dataset-mixing
link
Recently, Essential AI released a new 8B base model https://huggingface.co/EssentialAI/rnj-1 they highlighted the importance of data mix for pretraning -

"In the long run, we expect our methods to automatically represent, transform, and blend data to optimize measurable abilities in pre-training. Our work on modeling data taxonomies led to new approaches for jointly clustering and mixing data distributions under data repetition penalties. Many improvements in our STEM abilities can be traced back to this. "

This resonates with the recent work we did around optimal dataset mixing for pretraining where we saw have the right mix can increase the efficiency of training - 
https://huggingface.co/blog/codelion/optimal-dataset-mixing
🚀
codelion John6666 fei1212 Maani NNEngine
🔥
codelion unmodeled-tyler
👍
codelion