TopicVD: A Topic-Based Dataset of Video-Guided Multimodal Machine Translation for Documentaries

TL;DR

TopicVD advances documentary-level video-guided multimodal MT by providing a topic-annotated, context-preserving dataset and a cross-modal bidirectional attention model that fuses textual and visual signals. The dataset comprises 256 documentaries across 8 topics, 285 hours, and 122,930 CN-EN subtitle pairs, enabling domain-adaptive research and global-context modeling. Empirical results show that visual information improves translation over text-only MT, but performance degrades in out-of-domain settings, underscoring the need for domain-adaptation strategies; increasing contextual information further enhances translation. Overall, TopicVD highlights the importance of topic organization and global context in VMT and offers a scalable platform for researching documentary translation with multimodal signals.

Abstract

Most existing multimodal machine translation (MMT) datasets are predominantly composed of static images or short video clips, lacking extensive video data across diverse domains and topics. As a result, they fail to meet the demands of real-world MMT tasks, such as documentary translation. In this study, we developed TopicVD, a topic-based dataset for video-supported multimodal machine translation of documentaries, aiming to advance research in this field. We collected video-subtitle pairs from documentaries and categorized them into eight topics, such as economy and nature, to facilitate research on domain adaptation in video-guided MMT. Additionally, we preserved their contextual information to support research on leveraging the global context of documentaries in video-guided MMT. To better capture the shared semantics between text and video, we propose an MMT model based on a cross-modal bidirectional attention module. Extensive experiments on the TopicVD dataset demonstrate that visual information consistently improves the performance of the NMT model in documentary translation. However, the MMT model's performance significantly declines in out-of-domain scenarios, highlighting the need for effective domain adaptation methods. Additionally, experiments demonstrate that global context can effectively improve translation performance. % Dataset and our implementations are available at https://github.com/JinzeLv/TopicVD

Figures (3)

• Figure 1: Construction process of the TopicVD documentary translation dataset
• Figure 2: An overview of the proposed VMT model with cross-modal bidirectional attention
• Figure 3: Example of correct translation by the proposed method