IndicDLP: A Foundational Dataset for Multi-Lingual and Multi-Domain Document Layout Parsing

TL;DR

IndicDLP addresses the lack of large-scale, multilingual, multi-domain document layout data by introducing IndicDLP, a 119k-image collection spanning 12 document domains and 12 languages (11 Indic languages + English) with 42 region labels. The work includes a curated pretraining dataset (UED-mini) and demonstrates that English pretraining on IndicDLP enhances cross-domain layout parsing, while IndicDLP pretraining accelerates convergence and improves performance on other datasets. Experiments reveal strong domain- and language-generalization, with notable improvements over baselines and evidence of improved OCR and information extraction potential for Indic scripts. The dataset and accompanying resources are released to foster inclusive, high-performance document understanding across languages and domains.

Abstract

Document layout analysis is essential for downstream tasks such as information retrieval, extraction, OCR, and digitization. However, existing large-scale datasets like PubLayNet and DocBank lack fine-grained region labels and multilingual diversity, making them insufficient for representing complex document layouts. In contrast, human-annotated datasets such as M6Doc and D4LA offer richer labels and greater domain diversity, but are too small to train robust models and lack adequate multilingual coverage. This gap is especially pronounced for Indic documents, which encompass diverse scripts yet remain underrepresented in current datasets, further limiting progress in this space. To address these shortcomings, we introduce IndicDLP, a large-scale foundational document layout dataset spanning 11 representative Indic languages alongside English and 12 common document domains. Additionally, we curate UED-mini, a dataset derived from DocLayNet and M6Doc, to enhance pretraining and provide a solid foundation for Indic layout models. Our experiments demonstrate that fine-tuning existing English models on IndicDLP significantly boosts performance, validating its effectiveness. Moreover, models trained on IndicDLP generalize well beyond Indic layouts, making it a valuable resource for document digitization. This work bridges gaps in scale, diversity, and annotation granularity, driving inclusive and efficient document understanding.

Figures (15)

• Figure 1: Samples from the IndicDLP dataset highlighting its diversity across document formats, domains, languages, and temporal span. For improved differentiability, segmentation masks are used instead of bounding boxes to highlight regions more effectively. This figure is best viewed when zoomed in.
• Figure 2: Comparison of mAP Scores Across Different Domains for YOLOv10x, DocLayout-YOLO and RoDLA trained on IndicDLP.
• Figure 3: Qualitative analysis of YOLOv10x model predictions on IndicDLP test set when trained from scratch on DocLayNet (Column 1), D4LA (Column 2), M6Doc (Column 3), and IndicDLP (Column 4), compared to the Ground Truth (Column 5). For instance, in Rows 1 and 3, for M6Doc, despite having labels like advertisements and sidebars, distribution differences prevent accurate localization and classification. In Rows 2 and 4, models trained on IndicDLP show superior performance on non-standard and multicolumn layouts, respectively. Row 5 highlights partial figure detections for DocLayNet and D4LA, even though it is a common region across all the above datasets.
• Figure 4: Performance gap observed during finetuning on various datasets using YOLOv10x pretrained on IndicDLP compared to finetuning from scratch. The solid line represents finetuning with IndicDLP, while the dashed line represents finetuning without it. For M6Doc (center) and D4LA (right), pretraining on IndicDLP increases mAP by 2.8 points on average, while for DocLayNet (left), it leads to faster convergence.
• Figure 5: The above figure illustrates the contributions of 12 languages (left) and 12 document domains (right) in the IndicDLP dataset. The distribution is fairly balanced across both categories, with no single language or domain overwhelmingly dominating the dataset. This ensures a diverse and well-represented collection.