Exploring In-Image Machine Translation with Real-World Background

TL;DR

This work tackles In-Image Machine Translation under realistic conditions by introducing the complex-scenario IIMT task and the IIMT30k dataset, where real-world backgrounds accompany multi-font text. It presents DebackX, a three-stage pipeline that separates background from text-image, translates the text-image in isolation, and fuses the result back with the background to preserve visual fidelity and font consistency. Through extensive experiments against multiple baselines, DebackX demonstrates gains in both translation quality (BLEU/COMET) and visual realism (FID), aided by pre-training and font-adaptation studies. The approach advances practical IIMT by addressing error propagation, background integrity, and typography alignment, with implications for subtitle rendering and multimodal translation in the wild.

Abstract

In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another. Previous research on IIMT was primarily conducted on simplified scenarios such as images of one-line text with black font in white backgrounds, which is far from reality and impractical for applications in the real world. To make IIMT research practically valuable, it is essential to consider a complex scenario where the text backgrounds are derived from real-world images. To facilitate research of complex scenario IIMT, we design an IIMT dataset that includes subtitle text with real-world background. However previous IIMT models perform inadequately in complex scenarios. To address the issue, we propose the DebackX model, which separates the background and text-image from the source image, performs translation on text-image directly, and fuses the translated text-image with the background, to generate the target image. Experimental results show that our model achieves improvements in both translation quality and visual effect.

Figures (5)

• Figure 1: Illustration of simplified and complex scenario IIMT. Previous research mainly focuses on simplified scenario IIMT, which is far from reality. Exploring complex IIMT scenarios that are much closer to reality is necessary. We find that the current IIMT model cannot fully handle the complex scenario, since the translation lacks an integrated background and fails to maintain font consistency, leading to poor visual effect.
• Figure 2: Architecture of our proposed DebackX.
• Figure 3: Samples from the IIMT30k dataset with multiple fonts (Times New Roman, Arial, and Calibri).
• Figure 4: Detailed GPT-4o output of Case #1.
• Figure 5: Detailed GPT-4o output of Case #2.