Context-Informed Machine Translation of Manga using Multimodal Large Language Models

TL;DR

This work tackles automatic manga translation by leveraging multimodal large language models to exploit visual context and long-range narrative information. It systematically compares a range of translation approaches, from line-by-line to page-level and long-context methods, on JA-EN with OpenMantra data and a newly created JA-PL Love Hina dataset. The authors release an open-source manga translation evaluation suite and establish a 400-page JA-PL benchmark, reporting state-of-the-art performance for JA-EN and strong results for JA-PL, especially with page-by-page visual-context methods. Key findings include that visual context substantially improves translation, while longer contextual input yields mixed results, and they discuss broader implications for cross-lingual manga translation and future work with open models and multi-volume narratives.

Abstract

Due to the significant time and effort required for handcrafting translations, most manga never leave the domestic Japanese market. Automatic manga translation is a promising potential solution. However, it is a budding and underdeveloped field and presents complexities even greater than those found in standard translation due to the need to effectively incorporate visual elements into the translation process to resolve ambiguities. In this work, we investigate to what extent multimodal large language models (LLMs) can provide effective manga translation, thereby assisting manga authors and publishers in reaching wider audiences. Specifically, we propose a methodology that leverages the vision component of multimodal LLMs to improve translation quality and evaluate the impact of translation unit size, context length, and propose a token efficient approach for manga translation. Moreover, we introduce a new evaluation dataset -- the first parallel Japanese-Polish manga translation dataset -- as part of a benchmark to be used in future research. Finally, we contribute an open-source software suite, enabling others to benchmark LLMs for manga translation. Our findings demonstrate that our proposed methods achieve state-of-the-art results for Japanese-English translation and set a new standard for Japanese-Polish.

Figures (16)

• Figure 1: Comparison of translation outputs for methods with different context types. The preceding scene visually shows a TV, giving context to the complaints in purple. The previous and current scenes are set in a restaurant, making it improbable that "Suzume" refers to a sparrow rather than being a name. © Kira Ito
• Figure 2: A manga page: panel borders (green), example lines in speech bubbles (purple), free flowing text (orange) and sound effects (red). Courtesy of Akamatsu Ken, © Kodansha
• Figure 3: Fragment of a page annotated for the PBP-VIS-NUM method © Mitsuki Kuchitaka.
• Figure 4: Stages of the text detection pipeline. First, pixels belonging to letters are identified. Then, the pixels are clustered into utterances. Lastly, bounding boxes are computed. Courtesy of Akamatsu Ken, © Kodansha, from the Manga109-s dataset manga109_origmanga109_2manga109_building
• Figure 5: Page processing pipeline. The reading order is estimated based on the relative location of the detected panels (green) and text boxes (red). Courtesy of Akamatsu Ken, © Kodansha, from the Manga109-s dataset manga109_origmanga109_2manga109_building