Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages

Shaharukh Khan, Ali Faraz, Abhinav Ravi, Mohd Nauman, Mohd Sarfraz, Akshat Patidar, Raja Kolla, Chandra Khatri, Shubham Agarwal

Abstract

Multimodal research has predominantly focused on single-image reasoning, with limited exploration of multi-image scenarios. Recent models have sought to enhance multi-image understanding through large-scale pretraining on interleaved image-text datasets. However, most Vision-Language Models (VLMs) are trained primarily on English datasets, leading to inadequate representation of Indian languages. To address this gap, we introduce the Chitrakshara dataset series, covering 11 Indian languages sourced from Common Crawl. It comprises (1) Chitrakshara-IL, a large-scale interleaved pretraining dataset with 193M images, 30B text tokens, and 50M multilingual documents, and (2) Chitrakshara-Cap, which includes 44M image-text pairs with 733M tokens. This paper details the data collection pipeline, including curation, filtering, and processing methodologies. Additionally, we present a comprehensive quality and diversity analysis to assess the dataset's representativeness across Indic languages and its potential for developing more culturally inclusive VLMs.

Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages

Abstract

Multimodal research has predominantly focused on single-image reasoning, with limited exploration of multi-image scenarios. Recent models have sought to enhance multi-image understanding through large-scale pretraining on interleaved image-text datasets. However, most Vision-Language Models (VLMs) are trained primarily on English datasets, leading to inadequate representation of Indian languages. To address this gap, we introduce the Chitrakshara dataset series, covering 11 Indian languages sourced from Common Crawl. It comprises (1) Chitrakshara-IL, a large-scale interleaved pretraining dataset with 193M images, 30B text tokens, and 50M multilingual documents, and (2) Chitrakshara-Cap, which includes 44M image-text pairs with 733M tokens. This paper details the data collection pipeline, including curation, filtering, and processing methodologies. Additionally, we present a comprehensive quality and diversity analysis to assess the dataset's representativeness across Indic languages and its potential for developing more culturally inclusive VLMs.
Paper Structure (25 sections, 16 figures, 7 tables, 3 algorithms)

This paper contains 25 sections, 16 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Web crawled datasets
  4. Multimodal Interleaved datasets
  5. India-centric multilingual datasets
  6. Dataset: Chitrakshara
  7. HTML Pipeline
  8. Content Refinement Pipeline
  9. Multimodal Document Assembly
  10. Hierarchical Content Filtering
  11. Additional Heuristics
  12. Analysis
  13. Conclusion
  14. Data pipeline technical insights
  15. HTML Pipeline
  16. ...and 10 more sections

Figures (16)

  • Figure 1: Chitrakshara dataset creation pipeline
  • Figure 2: Illustration of multimodal document extraction from the web. On the left, Chitrakshara-Cap includes image alt-text pairs, while on the right, Chitrakshara-IL retains the interleaved structure (truncated) of text & images from the source Hindi document.
  • Figure 3: Domain distribution of the Chitrakshara-IL data.
  • Figure 4: Cumulative image count distribution per document.
  • Figure 5: Distribution of Documents Over Years.
  • ...and 11 more figures