Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora

TL;DR

Wasm tackles the scarcity of high-quality Arabic multimodal corpora by introducing a pipeline that processes Common Crawl data to produce interleaved multimodal outputs and Markdown-structured text-only corpora. It adapts OBELICS for Arabic, preserving document structure, interleaved text–image sequences, and DOM-level hierarchy, while implementing Arabic-specific perplexity modeling and fine-grained node-level deduplication. Key contributions include an end-to-end preprocessing framework, relaxed and language-aware tag filters, URL-based visual data handling, and a public dataset plus open-source components to enable reproducible research. The approach is positioned to improve pre-training for both LLMs and LMMs on Arabic data, with practical impact on model capabilities and multilingual NLP research.

Abstract

The performance of large language models (LLMs) and large multimodal models (LMMs) depends heavily on the quality and scale of their pre-training datasets. Recent research shows that large multimodal models trained on natural documents where images and text are interleaved outperform those trained only on image-text pairs across a wide range of benchmarks, leveraging advanced pre-trained models to enforce semantic alignment, image-sequence consistency, and textual coherence. For Arabic, however, the lack of high-quality multimodal datasets that preserve document structure has limited progress. In this paper, we present our pipeline Wasm for processing the Common Crawl dataset to create a new Arabic multimodal dataset that uniquely provides markdown output. Unlike existing Arabic corpora that focus solely on text extraction, our approach preserves the structural integrity of web content while maintaining flexibility for both text-only and multimodal pre-training scenarios. We provide a comprehensive comparative analysis of our data processing pipeline against those used for major existing datasets, highlighting the convergences in filtering strategies and justifying our specific design choices. To support future research, we publicly release a representative dataset dump along with the multimodal processing pipeline for Arabic.

Figures (3)

• Figure 1: Overview of our data processing pipeline.
• Figure 2: Structured output produced by our pipeline.
• Figure 3: detailed wasm pipline