Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video

Kevin Cai, Chonghua Liu, David M. Chan

TL;DR

The paper tackles the data bottleneck in automated video dubbing by introducing Anim-400K, a large-scale dataset of over 425K aligned JP–EN dubbed clips designed for end-to-end dubbing and a suite of secondary video tasks. It presents a bottom-up data collection and a top-down annotation pipeline to produce fully aligned audio, transcripts, speaker labels, and rich metadata, including backing tracks for realistic mixing. Beyond dubbing, Anim-400K supports video summarization, character identification, genre/theme/style classification, video quality analysis, and simultaneous translation, with EN subtitles overlapping JP audio to enable ST research. Publicly available and significantly larger than prior corpora, Anim-400K offers a valuable resource for advancing end-to-end dubbing research and related multimedia analysis, while also addressing ethical, cultural, and practical considerations inherent to large-scale synthetic audio generation.

Abstract

The Internet's wealth of content, with up to 60% published in English, starkly contrasts the global population, where only 18.8% are English speakers, and just 5.1% consider it their native language, leading to disparities in online information access. Unfortunately, automated processes for dubbing of video - replacing the audio track of a video with a translated alternative - remains a complex and challenging task due to pipelines, necessitating precise timing, facial movement synchronization, and prosody matching. While end-to-end dubbing offers a solution, data scarcity continues to impede the progress of both end-to-end and pipeline-based methods. In this work, we introduce Anim-400K, a comprehensive dataset of over 425K aligned animated video segments in Japanese and English supporting various video-related tasks, including automated dubbing, simultaneous translation, guided video summarization, and genre/theme/style classification. Our dataset is made publicly available for research purposes at https://github.com/davidmchan/Anim400K.

ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video

TL;DR

The paper tackles the data bottleneck in automated video dubbing by introducing Anim-400K, a large-scale dataset of over 425K aligned JP–EN dubbed clips designed for end-to-end dubbing and a suite of secondary video tasks. It presents a bottom-up data collection and a top-down annotation pipeline to produce fully aligned audio, transcripts, speaker labels, and rich metadata, including backing tracks for realistic mixing. Beyond dubbing, Anim-400K supports video summarization, character identification, genre/theme/style classification, video quality analysis, and simultaneous translation, with EN subtitles overlapping JP audio to enable ST research. Publicly available and significantly larger than prior corpora, Anim-400K offers a valuable resource for advancing end-to-end dubbing research and related multimedia analysis, while also addressing ethical, cultural, and practical considerations inherent to large-scale synthetic audio generation.

Abstract

The Internet's wealth of content, with up to 60% published in English, starkly contrasts the global population, where only 18.8% are English speakers, and just 5.1% consider it their native language, leading to disparities in online information access. Unfortunately, automated processes for dubbing of video - replacing the audio track of a video with a translated alternative - remains a complex and challenging task due to pipelines, necessitating precise timing, facial movement synchronization, and prosody matching. While end-to-end dubbing offers a solution, data scarcity continues to impede the progress of both end-to-end and pipeline-based methods. In this work, we introduce Anim-400K, a comprehensive dataset of over 425K aligned animated video segments in Japanese and English supporting various video-related tasks, including automated dubbing, simultaneous translation, guided video summarization, and genre/theme/style classification. Our dataset is made publicly available for research purposes at https://github.com/davidmchan/Anim400K.
Paper Structure (14 sections, 2 figures, 3 tables)

This paper contains 14 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction & Background
  2. Dataset
  3. Related Datasets:
  4. Data Collection:
  5. Annotation:
  6. Supported Secondary Tasks
  7. (3.1) Video Summarization/Teaser Generation:
  8. (3.2) Character Identification & Description:
  9. (3.3) Genre/Theme/Style Identification:
  10. (3.4) Video Quality Analysis:
  11. (3.5) Simultaneous Translation:
  12. Limitations & Ethics
  13. Conclusion
  14. References

Figures (2)

  • Figure 1: Anim-400K is a new dataset designed from the ground up for automated dubbing of video, and supporting a wide range of secondary video tasks ranging from simultaneous translation and guided video summarization to genre/theme/style classification.
  • Figure 2: Genres and themes present in the Anim-400K dataset.