Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval

Kyle Buettner, Adriana Kovashka

TL;DR

This work empirically shows performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German, and proposes and evaluates caption augmentation strategies.

Abstract

There is a scarcity of multilingual vision-language models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German. To address these gaps, we further propose and evaluate caption augmentation strategies. While we achieve mean recall improvements (+1.3), gaps still remain, indicating an open area of future work for the community.

Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval

TL;DR

This work empirically shows performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German, and proposes and evaluates caption augmentation strategies.

Abstract

There is a scarcity of multilingual vision-language models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German. To address these gaps, we further propose and evaluate caption augmentation strategies. While we achieve mean recall improvements (+1.3), gaps still remain, indicating an open area of future work for the community.
Paper Structure (8 sections, 1 figure, 3 tables)

This paper contains 8 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Experimental Methodology
  4. Benchmarking Details
  5. Methods Compared
  6. Key Findings
  7. Further Analysis
  8. Conclusion

Figures (1)

  • Figure 1: Example perception differences between native English and German speakers. Examples are captions from Flickr30K young-etal-2014-image and Multi30K elliott-etal-2016-multi30k. Note differences in mentioned objects ("sand arena", "parasol") and specificity ("Heurigen bench" vs. "table", "horse" vs. "bronco"). German captions here are translated to English.