Gaperon: A Peppered English-French Generative Language Model Suite
We release Gaperon, a fully open suite of French-English-coding language
models designed to advance transparency and reproducibility in large-scale
model training. The Gaperon family includes 1.5B, 8B, and 24B parameter models
trained on 2-4 trillion tokens, released with all elements of the training
pipeline: French and English datasets filtered with a neural quality
classifier, an efficient data curation and training framework, and hundreds of
intermediate checkpoints. Through this work, we study how data filtering and
contamination interact to shape both benchmark and generative performance. We
find that filtering for linguistic quality enhances text fluency and coherence
but yields subpar benchmark results, and that late deliberate contamination --
continuing training on data mixes that include test sets -- recovers
competitive scores while only reasonably harming generation quality. We discuss
how usual neural filtering can unintentionally amplify benchmark leakage. To
support further research, we also introduce harmless data poisoning during
pretraining, providing a realistic testbed for safety studies. By openly
releasing all models, datasets, code, and checkpoints, Gaperon establishes a
reproducible foundation for exploring the trade-offs between data curation,
evaluation, safety, and openness in multilingual language model development.
6903229772739622ee92a942
Nathan Godey
claimed_verified
2025-10-30T14:47:48.027Z
61d2d6a5291690e1c7b4dd2d
https://cdn-avatars.huggingface.co/v1/production/uploads/1666688353023-61d2d6a5291690e1c7b4dd2d.jpeg
Nathan Godey
nthngdy
user
nthngdy
6903229772739622ee92a943
Wissam Antoun
claimed_verified
2025-10-31T14:27:54.997Z
605350699008a64d7b43b78d
https://cdn-avatars.huggingface.co/v1/production/uploads/1616431735085-605350699008a64d7b43b78d.jpeg
Wissam Antoun
wissamantoun
user
wissamantoun
6903229772739622ee92a944
Rian Touchent
claimed_verified
2025-10-30T14:15:55.072Z
62a9b0acf6708cb85014f9dc
https://cdn-avatars.huggingface.co/v1/production/uploads/62a9b0acf6708cb85014f9dc/Sem1qcBt1lJjFEPK-xz4_.jpeg
Rian Touchent
rntc
user
rntc
6903229772739622ee92a945
Rachel Bawden
6903229772739622ee92a946
Éric de la Clergerie
6903229772739622ee92a947
Benoît Sagot
6903229772739622ee92a948
Djamé Seddah
Gaperon: A Peppered English-French Generative Language Model Suite
We release Gaperon, a fully open suite of French-English-coding language
models designed to advance transparency and reproducibility in large-scale
model training. The Gaperon family includes 1.5B, 8B, and 24B parameter models
trained on 2-4 trillion tokens, released with all elements of the training
pipeline: French and English datasets filtered with a neural quality
classifier, an efficient data curation and training framework, and hundreds of
intermediate checkpoints. Through this work, we study how data filtering and
contamination interact to shape both benchmark and generative performance. We
find that filtering for linguistic quality enhances text fluency and coherence
but yields subpar benchmark results, and that late deliberate contamination --
continuing training on data mixes that include test sets -- recovers
competitive scores while only reasonably harming generation quality. We discuss
how usual neural filtering can unintentionally amplify benchmark leakage. To
support further research, we also introduce harmless data poisoning during
pretraining, providing a realistic testbed for safety studies. By openly
releasing all models, datasets, code, and checkpoints, Gaperon establishes a
reproducible foundation for exploring the trade-offs between data curation,
evaluation, safety, and openness in multilingual language model development.
Gaperon, a suite of multilingual language models, advances transparency and reproducibility by releasing training pipelines, datasets, and checkpoints, and studies the impact of data filtering and contamination on model performance.
neural quality classifier data curation training framework intermediate checkpoints benchmark performance generative performance linguistic quality text fluency coherence benchmark leakage data poisoning safety studies