What Is Missing in Multilingual Visual Reasoning and How to Fix It

TL;DR

This work probes multilingual visual reasoning by contrasting proprietary systems (notably GPT-4V) with open models on NLVR2 and MaRVL, revealing strong cross-language performance for GPT-4V but gaps for open models in non-English contexts. It identifies three core challenges—multilinguality, complex reasoning, and multimodality—driving the need for targeted interventions. The authors propose Translate-Test, Visual Programming, and Reasoning with Captions, demonstrating substantial gains for open models in zero-shot settings and achieving state-of-the-art zero-shot results on MaRVL among open models. The findings underscore the potential to narrow the performance gap through modular reasoning, multilingual adaptation, and caption-based multimodal reasoning, with implications for more equitable multilingual AI systems.

Abstract

NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users. In this paper, we evaluate their multilingual, multimodal capabilities by testing on a visual reasoning task. We observe that proprietary systems like GPT-4V obtain the best performance on this task now, but open models lag in comparison. Surprisingly, GPT-4V exhibits similar performance between English and other languages, indicating the potential for equitable system development across languages. Our analysis on model failures reveals three key aspects that make this task challenging: multilinguality, complex reasoning, and multimodality. To address these challenges, we propose three targeted interventions including a translate-test approach to tackle multilinguality, a visual programming approach to break down complex reasoning, and a method that leverages image captioning to address multimodality. Our interventions achieve the best open performance on this task in a zero-shot setting, boosting open models LLaVA-v1.5-13B by 13.4%, LLaVA-v1.6-34B by 20.3%, and Qwen-VL by 16.7%, while also minorly improving GPT-4V's performance.

Figures (6)

• Figure 1: Our Contributions: First, we evaluate the multilingual visual reasoning abilities of various models; then, we analyze key challenges where models are falling short; lastly, we propose three interventions to address these challenges.
• Figure 2: Example from the MaRVL Dataset: Given two images and a statement, the task is to infer whether the statement is true or false of the given image pair.
• Figure 3: Performance of GPT-4V decreases as statement length increases.
• Figure 4: Flow chart visualizing the end-to-end testing in §\ref{['sec:results']} and all interventions performed in §\ref{['sec:intervention']}.
• Figure 5: VisProg example image pair.