Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling

Chaoqun Cui, Shijing Wang, Liangbin Huang, Qingqing Gu, Zhaolong Huang, Xiao Zeng, Wenji Mao

TL;DR

Her Hermes, an LLM-based automated subtitling framework that achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.

Abstract

Interlingual subtitling, which translates subtitles of visual media into a target language, is essential for entertainment localization but has not yet been explored in machine translation. Although Large Language Models (LLMs) have significantly advanced the general capabilities of machine translation, the distinctive characteristics of subtitle texts pose persistent challenges in interlingual subtitling, particularly regarding semantic coherence, pronoun and terminology translation, and translation expressiveness. To address these issues, we present Hermes, an LLM-based automated subtitling framework. Hermes integrates three modules: Speaker Diarization, Terminology Identification, and Expressiveness Enhancement, which effectively tackle the above challenges. Experiments demonstrate that Hermes achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.

Hermes the Polyglot: A Unified Framework to Enhance Expressiveness for Multimodal Interlingual Subtitling

TL;DR

Her Hermes, an LLM-based automated subtitling framework that achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.

Abstract

Interlingual subtitling, which translates subtitles of visual media into a target language, is essential for entertainment localization but has not yet been explored in machine translation. Although Large Language Models (LLMs) have significantly advanced the general capabilities of machine translation, the distinctive characteristics of subtitle texts pose persistent challenges in interlingual subtitling, particularly regarding semantic coherence, pronoun and terminology translation, and translation expressiveness. To address these issues, we present Hermes, an LLM-based automated subtitling framework. Hermes integrates three modules: Speaker Diarization, Terminology Identification, and Expressiveness Enhancement, which effectively tackle the above challenges. Experiments demonstrate that Hermes achieves state-of-the-art diarization performance and generates expressive, contextually coherent translations, thereby advancing research in interlingual subtitling.
Paper Structure (42 sections, 6 equations, 9 figures, 11 tables, 2 algorithms)

This paper contains 42 sections, 6 equations, 9 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Task Definition and Notations
  3. Proposed Framework
  4. Speaker Diarization
  5. Overview
  6. Feature Clustering
  7. Speaker Registration
  8. Speaker Supplementation
  9. Terminology Identification
  10. Overview
  11. Terminology LLM Training
  12. Expressiveness Enhancement
  13. Overview
  14. Translation Model Training
  15. Sampling Strategy
  16. ...and 27 more sections

Figures (9)

  • Figure 1: The overall structure of the unified framework Hermes to enhance expressiveness for interlingual subtitling.
  • Figure 2: Speaker registration.
  • Figure 3: A zh$\Rightarrow$en terminology example. "钦天监” is an ancient Chinese institution responsible for astronomical observations, calendar compilation, and weather prediction.
  • Figure 4: Impact of sampling size on translation quality.
  • Figure 5: Impact of model size on translation quality.
  • ...and 4 more figures