Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.
text
Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.
new_line
new_line
🤗 corpus v1: 
text
🤗 corpus v1: 
https://huggingface.co/datasets/cis-lmu/GlotCC-V1
cis-lmu/GlotCC-V1
dataset
resource
https://huggingface.co/datasets/cis-lmu/GlotCC-V1
new_line
🐱 pipeline v3: 
text
🐱 pipeline v3: 
https://github.com/cisnlp/GlotCC
https://github.com/cisnlp/GlotCC
link
new_line
new_line
More details? Stay tuned for our upcoming paper.
text
More details? Stay tuned for our upcoming paper.
new_line
More data? In the next version, we plan to include additional snapshots of CommonCrawl.
text
More data? In the next version, we plan to include additional snapshots of CommonCrawl.
new_line
new_line
Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).
text
Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).
new_line
Introducing GlotCC: a new 2TB corpus based on an early 2024 CommonCrawl snapshot with data for 1000+ languages.

🤗 corpus v1: https://huggingface.co/datasets/cis-lmu/GlotCC-V1
🐱 pipeline v3: https://github.com/cisnlp/GlotCC

More details? Stay tuned for our upcoming paper.
More data? In the next version, we plan to include additional snapshots of CommonCrawl.

Limitation: Due to the lower frequency of low-resource languages compared to others, there are sometimes only a few sentences available for very low-resource languages. However, the data volume for English in this version stands at 750GB, and the top 200 languages still have a strong presence in our data (see plot attached; we write the index for every 20 languages, meaning the 10th index is the 200th language).

👍
kargaranamir osanseviero eliebak yjernite ayymen Jakh0103