Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages

Swastik R

Abstract

Vision-language models score well on mathematical, scientific, and spatial reasoning benchmarks, yet these evaluations are overwhelmingly English. I present the first cross-lingual visual reasoning audit for Indian languages. 980 questions from MathVista, ScienceQA, and MMMU are translated into Hindi, Tamil, Telugu, Bengali, Kannada, and Marathi using IndicTrans2, with Gemini 2.0 Flash cross-verification on 50 samples per language (inter-translator agreement 0.79-0.84). Eight VLMs, from 7B open-source models to GPT-4o, are evaluated across all seven languages, yielding 68,600 inference records that include text-only and chain-of-thought ablations. I find accuracy drops of 9.8-25 percentage points when switching from English to an Indian language, with Dravidian languages suffering up to 13.2 pp more than Indo-Aryan. Chain-of-thought prompting degrades Bengali (-14.4 pp) and Kannada (-11.4 pp) rather than helping, exposing English-centric reasoning chains. Aya-Vision-8B, built for 23 languages, still drops 28.5 pp on Dravidian scripts; multilingual pretraining alone does not transfer visual reasoning. I release the translated benchmark and all model outputs.

Do Multilingual VLMs Reason Equally? A Cross-Lingual Visual Reasoning Audit for Indian Languages

Abstract

Vision-language models score well on mathematical, scientific, and spatial reasoning benchmarks, yet these evaluations are overwhelmingly English. I present the first cross-lingual visual reasoning audit for Indian languages. 980 questions from MathVista, ScienceQA, and MMMU are translated into Hindi, Tamil, Telugu, Bengali, Kannada, and Marathi using IndicTrans2, with Gemini 2.0 Flash cross-verification on 50 samples per language (inter-translator agreement 0.79-0.84). Eight VLMs, from 7B open-source models to GPT-4o, are evaluated across all seven languages, yielding 68,600 inference records that include text-only and chain-of-thought ablations. I find accuracy drops of 9.8-25 percentage points when switching from English to an Indian language, with Dravidian languages suffering up to 13.2 pp more than Indo-Aryan. Chain-of-thought prompting degrades Bengali (-14.4 pp) and Kannada (-11.4 pp) rather than helping, exposing English-centric reasoning chains. Aya-Vision-8B, built for 23 languages, still drops 28.5 pp on Dravidian scripts; multilingual pretraining alone does not transfer visual reasoning. I release the translated benchmark and all model outputs.

Paper Structure

This paper contains 36 sections, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Indian-language VLM benchmarks.
  4. Multilingual VLM evaluation beyond India.
  5. Visual reasoning benchmarks.
  6. Multilingual reasoning in LLMs.
  7. Indian NLP resources.
  8. Methodology
  9. Dataset Construction
  10. Languages
  11. Translation
  12. Models
  13. Evaluation Protocol
  14. Results
  15. Overall Accuracy
  16. ...and 21 more sections

Figures (10)

  • Figure 1: Accuracy heatmap across eight models and seven languages. Every model degrades from English (left column) to Indian languages, with Dravidian languages (ta, te, kn) consistently darker than Indo-Aryan (hi, bn, mr).
  • Figure 2: Average accuracy drop from English, per language, averaged across models. Dravidian languages (ta, te, kn) consistently suffer larger drops than Indo-Aryan (hi, bn, mr).
  • Figure 3: Per-dataset accuracy for English versus the average across Indian languages. MathVista shows the largest drops for most models. Gemma 3-27B achieves a negative drop on ScienceQA (Indian languages outperform English).
  • Figure 4: Radar chart of accuracy per source dataset for each language (averaged over all models). English sets the outer reference; Dravidian languages (Tamil, Telugu, Kannada) cluster closer to the centre on MathVista and MMMU, confirming that reasoning-heavy tasks suffer the most.
  • Figure 5: Cross-lingual consistency: percentage of questions receiving the same extracted answer across all languages with valid extractions ($\geq$4 of 7).
  • ...and 5 more figures