VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages

TL;DR

VLURes introduces a comprehensive multilingual benchmark to evaluate vision-language understanding across English, Japanese, Swahili, and Urdu using long, article-length textual context paired with images. It encompasses eight vision-language tasks, including a novel Unrelatedness task that tests the model’s ability to ignore irrelevant text. The benchmark combines automatic evaluation with native-language human judgments and demonstrates substantial cross-lingual gaps, with GPT-4o achieving the best performance but still lagging behind humans, while open-source models show more pronounced deficits except after finetuning. A key takeaway is that prompting with rationales helps, but gains are uneven across languages; the work underscores the need for multilingual pretraining and targeted fine-tuning to advance multilingual intelligent agents in real-world, multi-modal settings.

Abstract

Vision Language Models (VLMs) are pivotal for advancing perception in intelligent agents. Yet, evaluation of VLMs remains limited to predominantly English-centric benchmarks in which the image-text pairs comprise short texts. To evaluate VLM fine-grained abilities, in four languages under long-text settings, we introduce a novel multilingual benchmark VLURes featuring eight vision-and-language tasks, and a pioneering unrelatedness task, to probe the fine-grained Visual and Linguistic Understanding capabilities of VLMs across English, Japanese, and low-resource languages, Swahili, and Urdu. Our datasets, curated from web resources in the target language, encompass ten diverse image categories and rich textual context, introducing valuable vision-language resources for Swahili and Urdu. By prompting VLMs to generate responses and rationales, evaluated automatically and by native speakers, we uncover performance disparities across languages and tasks critical to intelligent agents, such as object recognition, scene understanding, and relationship understanding. We conducted evaluations of ten VLMs with VLURes. The best performing model, GPT-4o, achieves an overall accuracy of 90.8% and lags human performance by 6.7%, though the gap is larger for open-source models. The gap highlights VLURes' critical role in developing intelligent agents to tackle multi-modal visual reasoning.

Figures (16)

• Figure 1: VLURes consists of eight tasks shown above. The tasks enable the intelligent agent to understand its surroundings. We evaluate Vision Language Models' performance on VLURes. It offers image-text pairs in En, Jp, Sw, Ur, and each image is embedded in article-length prose.
• Figure 2: VLURes development process. Top: We selected the VL tasks and constructed datasets for each task and language. We leverage the image-text pairs as input to the VLMs and prompt the VLMs to generate responses for each task. Lastly, we deploy both automatic evaluation using LLM-as-a-Judge approach and hire native speakers to judge VLM-response quality manually. Bottom: We chose the tasks in VLURes to compare two kinds of visual reasoning for each image-text pair in the input: image-only reasoning (shown in (a)) and image+text reasoning (shown in (b)).
• Figure 3: Statistics of VLURes.
• Figure 6: Comparison of VLM performance on eight VL tasks under the One-shot, With Rationales setting. Left: English Input/Output. Right: Japanese Input/Output.
• Figure 7: Comparison of VLM performance on eight VL tasks under the One-shot, With Rationales setting. Left: Swahili Input/Output. Right: Urdu Input/Output.