Why do LLaVA Vision-Language Models Reply to Images in English?

TL;DR

This work uncovers an Image-induced Fidelity Loss ($IFL$) in LLaVA-style vision-language models, where adding an image to a query biases responses toward English across languages. Through large-scale ablations of design choices and targeted mechanistic analyses of internal representations, the authors locate the bias primarily in the language backbone and show that switching to bilingual backbones reduces $IFL$, while visual inputs reside in a distinct embedding space from text. A simple at-runtime intervention that steers intermediate-layer representations toward the target language yields substantial fidelity gains, validating a mechanistic path to mitigation. The findings highlight the need for multilingual emphasis in multimodal architectures and propose practical strategies to build more inclusive vision-language systems for non-English contexts.

Abstract

We uncover a surprising multilingual bias occurring in a popular class of multimodal vision-language models (VLMs). Including an image in the query to a LLaVA-style VLM significantly increases the likelihood of the model returning an English response, regardless of the language of the query. This paper investigates the causes of this loss with a two-pronged approach that combines extensive ablation of the design space with a mechanistic analysis of the models' internal representations of image and text inputs. Both approaches indicate that the issue stems in the language modelling component of the LLaVA model. Statistically, we find that switching the language backbone for a bilingual language model has the strongest effect on reducing this error. Mechanistically, we provide compelling evidence that visual inputs are not mapped to a similar space as text ones, and that intervening on intermediary attention layers can reduce this bias. Our findings provide important insights to researchers and engineers seeking to understand the crossover between multimodal and multilingual spaces, and contribute to the goal of developing capable and inclusive VLMs for non-English contexts.

Figures (3)

• Figure 1: Effect of adding image to query on response fidelity with 95% confidence intervals. All estimates are aggregated over languages within the benchmark and debiased using the DSL estimator.
• Figure 2: (LEFT): UMAP visualization of image and text embeddings from a multimodal language model. Image embeddings are shown clustering distinctly from text embeddings, indicating a unique separation in the latent space. This segregation highlights potential areas of focus for addressing fidelity loss in multimodal communication. (Center and Right):Centered Kernel Alignment (CKA) heatmap showing the similarity of vision embeddings across two differently trained language models. CKA based on a linear kernel is shown in the lower triangle; CKA based on an RBF kernel is shown in the upper triangle. The heatmap reveals a high degree of similarity in how vision data is embedded, regardless of the language model’s architecture or training data specifics. This uniformity suggests that the method of integrating visual data into language models is a critical factor affecting fidelity.
• Figure 3: Average fidelity by model and eval for query with and without Images