Wasm: A Pipeline for Constructing Structured Arabic Interleaved
  Multimodal Corpora
The performance of large language models (LLMs) and large multimodal models
(LMMs) depends heavily on the quality and scale of their pre-training datasets.
Recent research shows that large multimodal models trained on natural documents
where images and text are interleaved outperform those trained only on
image-text pairs across a wide range of benchmarks, leveraging advanced pre-
trained models to enforce semantic alignment, image-sequence consistency, and
textual coherence. For Arabic, however, the lack of high-quality multimodal
datasets that preserve document structure has limited progress. In this paper,
we present our pipeline Wasm for processing the Common Crawl dataset to create
a new Arabic multimodal dataset that uniquely provides markdown output. Unlike
existing Arabic corpora that focus solely on text extraction, our approach
preserves the structural integrity of web content while maintaining flexibility
for both text-only and multimodal pre-training scenarios. We provide a
comprehensive comparative analysis of our data processing pipeline against
those used for major existing datasets, highlighting the convergences in
filtering strategies and justifying our specific design choices. To support
future research, we publicly release a representative dataset dump along with
the multimodal processing pipeline for Arabic.
6912c6cba644ba07c499c7dd
Khalil Hennara
claimed_verified
2025-11-11T19:43:31.158Z
65276c7911a8a521c91bc10f
https://cdn-avatars.huggingface.co/v1/production/uploads/65276c7911a8a521c91bc10f/39dbuUtqQTJERTJ_WkxSh.jpeg
Khalil Hennara
Hennara
user
Hennara
6912c6cba644ba07c499c7de
Ahmad Bastati
claimed_verified
2025-11-17T10:31:56.092Z
662a4615cf541c92cf0062b9
https://cdn-avatars.huggingface.co/v1/production/uploads/662a4615cf541c92cf0062b9/9sPA5-HCCdpgk0RgDM1nk.jpeg
Ahmad Bastati
Bastati
user
Bastati
6912c6cba644ba07c499c7df
Muhammad Hreden
claimed_verified
2025-11-12T12:21:09.582Z
6496df4b3c64d75523a11973
https://cdn-avatars.huggingface.co/v1/production/uploads/6496df4b3c64d75523a11973/F54H8I0SVpBZMtA9iS1R_.jpeg
Muhammad Hreden
muhammad0-0hreden
user
muhammad0-0hreden
6912c6cba644ba07c499c7e0
Mohamed Motasim Hamed
6912c6cba644ba07c499c7e1
Zeina Aldallal
claimed_verified
2025-11-12T12:21:07.736Z
65704741e1cfce1764ce652e
/avatars/9189aaf417426af4ebe381ed364a6c0e.svg
Zeina Aldallal
ZeinaD
user
ZeinaD
6912c6cba644ba07c499c7e2
Sara Chrouf
6912c6cba644ba07c499c7e3
Safwan AlModhayan
Wasm: A Pipeline for Constructing Structured Arabic Interleaved
  Multimodal Corpora
The performance of large language models (LLMs) and large multimodal models
(LMMs) depends heavily on the quality and scale of their pre-training datasets.
Recent research shows that large multimodal models trained on natural documents
where images and text are interleaved outperform those trained only on
image-text pairs across a wide range of benchmarks, leveraging advanced pre-
trained models to enforce semantic alignment, image-sequence consistency, and
textual coherence. For Arabic, however, the lack of high-quality multimodal
datasets that preserve document structure has limited progress. In this paper,
we present our pipeline Wasm for processing the Common Crawl dataset to create
a new Arabic multimodal dataset that uniquely provides markdown output. Unlike
existing Arabic corpora that focus solely on text extraction, our approach
preserves the structural integrity of web content while maintaining flexibility
for both text-only and multimodal pre-training scenarios. We provide a
comprehensive comparative analysis of our data processing pipeline against
those used for major existing datasets, highlighting the convergences in
filtering strategies and justifying our specific design choices. To support
future research, we publicly release a representative dataset dump along with
the multimodal processing pipeline for Arabic.
A pipeline for processing the Common Crawl dataset to create a new Arabic multimodal dataset that preserves document structure and supports both text-only and multimodal pre-training.
large language models large multimodal models pre-training datasets natural documents semantic alignment image-sequence consistency textual coherence Arabic multimodal datasets markdown output web content data processing pipeline comparative analysis dataset dump