Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation

TL;DR

The paper tackles in-image machine translation by proposing Translatotron-V(ision), an end-to-end model that generates RGB translated images. It introduces four modules (image encoder, target text decoder, image decoder, image tokenizer) to reduce modeling burden and avoid predicting raw pixels, aided by a two-stage training framework and multi-task learning (OCR and TIT) plus knowledge distillation from a text-to-image teacher. A new evaluation metric, Structure-BLEU, accounts for text localization within images. Empirical results show competitive performance with cascaded models while using only 70.9% of their parameters and outperforming pixel-level end-to-end baselines, demonstrating practicality for real-world IIMT tasks.

Abstract

In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this regard, conventional cascaded methods suffer from issues such as error propagation, massive parameters, and difficulties in deployment and retaining visual characteristics of the input image. Thus, constructing end-to-end models has become an option, which, however, faces two main challenges: 1) the huge modeling burden, as it is required to simultaneously learn alignment across languages and preserve the visual characteristics of the input image; 2) the difficulties of directly predicting excessively lengthy pixel sequences. In this paper, we propose \textit{Translatotron-V(ision)}, an end-to-end IIMT model consisting of four modules. In addition to an image encoder, and an image decoder, our model contains a target text decoder and an image tokenizer. Among them, the target text decoder is used to alleviate the language alignment burden, and the image tokenizer converts long sequences of pixels into shorter sequences of visual tokens, preventing the model from focusing on low-level visual features. Besides, we present a two-stage training framework for our model to assist the model in learning alignment across modalities and languages. Finally, we propose a location-aware evaluation metric called Structure-BLEU to assess the translation quality of the generated images. Experimental results demonstrate that our model achieves competitive performance compared to cascaded models with only 70.9\% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.

Figures (10)

• Figure 1: The illustration of two paradigms of IIMT.
• Figure 2: Overview of our model.
• Figure 3: Overview of our two-stage training framework. The modules enclosed by dotted lines will be removed during inference.
• Figure 4: Translation examples of different models on the De$\rightarrow$En IIMT task.
• Figure 5: BLEU scores on different groups divided according to WER. We compute the WER using the text produced by the OCR toolkit.