These past months, I've been busy baking a special sort of Croissant 🥐 with an awesome team ! 
text
These past months, I've been busy baking a special sort of Croissant 🥐 with an awesome team ! 
new_line
new_line
🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models ! 
text
🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models ! 
new_line
new_line
💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)... 
text
💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)... 
new_line
new_line
⚖️ Assessing  LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !
text
⚖️ Assessing  LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !
new_line
new_line
🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !
text
🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !
new_line
new_line
🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?
text
🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?
new_line
new_line
Many more things to say, for those interested, I recommend checking out:
text
Many more things to say, for those interested, I recommend checking out:
new_line
new_line
🗞️ The blogpost: 
text
🗞️ The blogpost: 
https://huggingface.co/blog/manu/croissant-llm-blog
https://huggingface.co/blog/manu/croissant-llm-blog
link
new_line
📖 The 45 page report with lots of gems: 
text
📖 The 45 page report with lots of gems: 
https://arxiv.org/abs/2402.00786
https://arxiv.org/abs/2402.00786
link
new_line
🤖 Models, Data, Demo: 
text
🤖 Models, Data, Demo: 
https://cdn-avatars.huggingface.co/v1/production/uploads/60f2e021adf471cbdf8bb660/PuLzi_Evn-UPfTcZP0DSU.png
https://huggingface.co/croissantllm
croissantllm
org
resource
https://huggingface.co/croissantllm
new_line
These past months, I've been busy baking a special sort of Croissant 🥐 with an awesome team ! 

🥐 CroissantLLM is a truly bilingual language model trained on 3 trillion tokens of French and English data. In its size category (<2B), it is the best model in French, but it also rivals the best monolingual English models ! 

💾 To train it, we collected, filtered and cleaned huge quantities of permissively licensed French data, across various domains (legal, administrative, cultural, scientific), and different text modalities (speech transcriptions, movie subtitles, encyclopedias, forums, webpages)... 

⚖️ Assessing  LLM performance is not easy, especially outside of English, and to this end we crafted a novel evaluation benchmark, FrenchBench, aiming to assess reasoning, factual knowledge, and linguistic capabilities of models in French !

🔎 The best current LLMs are hidden behind a shroud of mystery, trained with undisclosed training data mixes or strategies. We go the opposite way, releasing all of the project's artefacts (model checkpoints, data, training details, evaluation benchmarks...) We obtain 81 % of the Stanford FMTI transparency criterias, far ahead of even most open initiatives !

🧪Beyond a powerful industrial resource, our transparent initiative is a stepping stone for many scientific questions ! How does teaching a model two languages instead of one splits its monolingual ability ? Does training on so much French help the model integrate French-centric knowledge and cultural biases ? How does the model memorize the training data ?

Many more things to say, for those interested, I recommend checking out:

🗞️ The blogpost: https://huggingface.co/blog/manu/croissant-llm-blog
📖 The 45 page report with lots of gems: https://arxiv.org/abs/2402.00786
🤖 Models, Data, Demo: https://huggingface.co/croissantllm

❤️
eliolio gsarti euclaise eliebak kramp yozozaya clem blanchon victor mvaloatto osanseviero samusenps santiviquez ksiabani batmac nouamanetazi davanstrien fffiloni alielfilali01 sbrandeis Soubz velaia
🤯
fffiloni Soubz
5e7749883d77a72421292d07
https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg
Gabriele Sarti
gsarti
user
5e67bdd61009063689407479
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857146757-5e67bdd61009063689407479.jpeg
Clem 🤗
clem
user
64955109ac70da05b7aacb9a
https://cdn-avatars.huggingface.co/v1/production/uploads/64955109ac70da05b7aacb9a/bZKEz24ZfaWDSI33yHUmR.png
Kostas Siabanis
ksiabani
user