Multimodal Machine Translation with Visual Scene Graph Pruning

TL;DR

This work tackles visual information redundancy in multimodal machine translation by introducing Visual Scene Graph Pruning (PSG), which uses a language scene graph to guide pruning of the visual scene graph before translation. The model encodes both visual G_v and language G_l information in a Transformer-based encoder, applies a multi-step pruning strategy with KL constraints, and optimizes with a combined objective L = L_mmt + L_prune + L_nmt, where a text-only NMT loss helps stabilize multimodal learning. Extensive experiments on Multi30K, AmbigCaps, and CoMMuTE show PSG achieves state-of-the-art or competitive results across BLEU, METEOR, COMET, and Accuracy metrics, demonstrating effective noise reduction and improved disambiguation. The findings highlight the value of structural pruning of visual information guided by linguistic context for more faithful and robust multimodal translation, with practical implications for efficient cross-modal fusion.

Abstract

Multimodal machine translation (MMT) seeks to address the challenges posed by linguistic polysemy and ambiguity in translation tasks by incorporating visual information. A key bottleneck in current MMT research is the effective utilization of visual data. Previous approaches have focused on extracting global or region-level image features and using attention or gating mechanisms for multimodal information fusion. However, these methods have not adequately tackled the issue of visual information redundancy in MMT, nor have they proposed effective solutions. In this paper, we introduce a novel approach--multimodal machine translation with visual Scene Graph Pruning (PSG), which leverages language scene graph information to guide the pruning of redundant nodes in visual scene graphs, thereby reducing noise in downstream translation tasks. Through extensive comparative experiments with state-of-the-art methods and ablation studies, we demonstrate the effectiveness of the PSG model. Our results also highlight the promising potential of visual information pruning in advancing the field of MMT.

Figures (6)

• Figure 1: Visual scene graph pruning with guidance from language scene graphs.
• Figure 2: Overview of the PSG. The PSG framework consists of five key components: text sequence tokenization and embedding, scene graph extraction, scene graph pruning, joint representation encoding, and text decoding.
• Figure 3: Comparison of attention visualization before and after pruning.
• Figure 4: Sensitivity analysis results concerning the training batch size on Multi30K English-German benchmark.
• Figure 5: Sensitivity analysis results concerning the training batch size on Multi30K English-French benchmark.