DuwatBench: Bridging Language and Visual Heritage through an Arabic Calligraphy Benchmark for Multimodal Understanding

TL;DR

This work introduces DuwatBench, a 1.27K-sample Arabic calligraphy benchmark spanning six styles with transcriptions and detection annotations to evaluate vision–language models on visually complex text. It provides a rigorous dataset taxonomy, multi-tier validation, and bounding-box annotations to enable realistic recognition and localization assessments. A broad evaluation across 13 Arabic and multilingual systems reveals that ornate scripts with dense ligatures and diacritics pose substantial challenges, with bounding-box grounding and larger multilingual models offering notable robustness. By releasing public datasets and baselines, the study advances culturally grounded AI for heritage preservation, education, and digital humanities through script-aware, inclusive evaluation.

Abstract

Arabic calligraphy represents one of the richest visual traditions of the Arabic language, blending linguistic meaning with artistic form. Although multimodal models have advanced across languages, their ability to process Arabic script, especially in artistic and stylized calligraphic forms, remains largely unexplored. To address this gap, we present DuwatBench, a benchmark of 1,272 curated samples containing about 1,475 unique words across six classical and modern calligraphic styles, each paired with sentence-level detection annotations. The dataset reflects real-world challenges in Arabic writing, such as complex stroke patterns, dense ligatures, and stylistic variations that often challenge standard text recognition systems. Using DuwatBench, we evaluated 13 leading Arabic and multilingual multimodal models and showed that while they perform well on clean text, they struggle with calligraphic variation, artistic distortions, and precise visual-text alignment. By publicly releasing DuwatBench and its annotations, we aim to advance culturally grounded multimodal research, foster fair inclusion of the Arabic language and visual heritage in AI systems, and support continued progress in this area. Our dataset (https://huggingface.co/datasets/MBZUAI/DuwatBench) and evaluation suit (https://github.com/mbzuai-oryx/DuwatBench) are publicly available.

Figures (9)

• Figure 1: DuwatBench Taxonomy and Distribution. DuwatBench encompasses six principal Arabic calligraphy styles (Thuluth, Diwani, Kufic, Naskh, Ruq'ah, and Nasta'liq). The benchmark includes a distribution of collected samples ranging from 706 in Thuluth to 67 in Nasta'liq. Beyond style diversity, the taxonomy incorporates categories, including non-religious terms, Quranic words, devotional expressions and hadith, names of the Prophet and companions, names of Allah, and person or place names. This organization provides a structured basis for evaluating both the visual variation and semantic depth of Arabic calligraphy.
• Figure 2: Examples of DuwatBench Calligraphic Styles. Representative samples from the six calligraphic styles in DuwatBench: Thuluth, Diwani, Kufic, Ruq'ah, Naskh, and Nasta'liq. Each entry displays the artwork with bounding box annotations, transcription, and metadata such as style, text content, theme, and word count. These examples highlight the dataset’s diversity in structure, composition, and artistic context, spanning both religious and non-religious inscriptions.
• Figure 3: Data collection and verification pipeline for DuwatBench. The process begins with sourcing candidate images from digital archives and community repositories, followed by quality-based selection and manual transcription by native Arabic speakers. Each sample is denoised, normalized, and assigned bounding boxes with text–style and theme annotations. A multi-tier validation stage involving quality control, stylistic checks, and expert review ensures both textual accuracy and calligraphic fidelity. Final samples are classified and aggregated by style and thematic category, forming the curated DuwatBench corpus.
• Figure 4: Annotation Workflow in DuwatBench. The figure illustrates the end-to-end annotation process followed in DuwatBench, from visual quality screening and bounding box drawing to transcription, style labeling, and word-count generation. Each stage ensures completeness, legibility, and consistent metadata alignment between text and visual form, supporting accurate multimodal evaluation.
• Figure 5: Qualitative Comparison of Model Outputs Across Calligraphic Styles Examples from DuwatBench showing transcription quality across six calligraphic styles, evaluated using 13 open and closed-source OCR and vision-language models. Each column presents the model’s predicted text aligned with the ground truth. The figure highlights challenges such as stroke density, curved baselines, and complex letter connections that complicate recognition, particularly in Thuluth, Diwani, and Kufic scripts. Note: Cells highlighted in green indicate correct predictions where diacritic variations are ignored; inaccurate or misplaced hamza marks are considered errors.