Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering

TL;DR

This work uncovers translation artifacts arising from machine translation in cross-lingual VQA, showing that distribution shifts between training and evaluation degrade generalization when using translate-test. By evaluating a broad set of multilingual and monolingual VL models across 13 languages and multiple MT systems, the study demonstrates that training on MT-translated data can yield modest gains, and that artifact-driven differences manifest in model representations and lexical properties. The authors propose lightweight data augmentation strategies, MERGE and TAG, to mitigate artifacts by mixing human and MT data and signaling translation origin, which improves performance on both English and translated evaluation sets. Overall, the findings highlight a practical path to more robust multilingual VQA systems and motivate further exploration of translation dynamics in multimodal learning.

Abstract

Building a reliable visual question answering~(VQA) system across different languages is a challenging problem, primarily due to the lack of abundant samples for training. To address this challenge, recent studies have employed machine translation systems for the cross-lingual VQA task. This involves translating the evaluation samples into a source language (usually English) and using monolingual models (i.e., translate-test). However, our analysis reveals that translated texts contain unique characteristics distinct from human-written ones, referred to as translation artifacts. We find that these artifacts can significantly affect the models, confirmed by extensive experiments across diverse models, languages, and translation processes. In light of this, we present a simple data augmentation strategy that can alleviate the adverse impacts of translation artifacts.

Figures (16)

• Figure 1: Predictions of LXMERT LXMERT on the original (left) and translated (right) questions. The model is correct for the human-written question but is incorrect for the correctly translated one. The original Korean question is "이 동물들은 모두 같은 종입니까?". For model visualization, we use an attention-based method by Attentionvisualization.
• Figure 2: Averaged multilingual models results The en* and avg. denote the RT-translated English evaluation set and the averaged cross-lingual transfer results, respectively. Full results of each multilingual model are in Fig. \ref{['fig:multilingual_full']}.
• Figure 3: A distribution of different translation errors in sampled questions from Korean translate-test set.
• Figure 4: A model is accurate for the original human-written question, but fails for a translated one. The Original Korean question is "그 사람이 전화 통화를 하고 있습니까?". Further annotation results are in Fig. \ref{['fig:case_study_results']}.
• Figure 5: Representation discrepancy of translate-test evaluation samples against training samples from different data origins (Human and MT). Pretrained or finetuned VisualBERT is used to encode representation, and FID is used as a distance metric. A lower score indicates a low distance between training and evaluation samples. Full results across different languages are in Fig. \ref{['fig:fid']}.