Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dialogue Understandability: Why are we streaming movies with subtitles?

Helard Becerra Martinez, Alessandro Ragano, Diptasree Debnath, Asad Ullah, Crisron Rudolf Lucas, Martin Walsh, Andrew Hines

TL;DR

This work introduces Dialogue Understandability (DU) as a holistic measure of how viewers follow movie plots under streaming conditions, extending beyond traditional intelligibility to encompass cognitive effort and QoE. It proposes a working definition, a six-factor root-cause framework, and a mapping to QoE and the movie lifecycle, then evaluates existing objective tools through pilot experiments to assess their suitability for DU. The paper highlights gaps in measurement maturity, argues for adaptation and multi-source modeling (audio, video, subtitles, metadata), and discusses practical avenues such as dialogue enhancement and smart subtitling. Its contributions pave the way for targeted datasets and objective tools to predict and improve DU, with implications for accessibility, localization, and user experience in streaming services.

Abstract

Watching movies and TV shows with subtitles enabled is not simply down to audibility or speech intelligibility. A variety of evolving factors related to technological advances, cinema production and social behaviour challenge our perception and understanding. This study seeks to formalise and give context to these influential factors under a wider and novel term referred to as Dialogue Understandability. We propose a working definition for Dialogue Understandability being a listener's capacity to follow the story without undue cognitive effort or concentration being required that impacts their Quality of Experience (QoE). The paper identifies, describes and categorises the factors that influence Dialogue Understandability mapping them over the QoE framework, a media streaming lifecycle, and the stakeholders involved. We then explore available measurement tools in the literature and link them to the factors they could potentially be used for. The maturity and suitability of these tools is evaluated over a set of pilot experiments. Finally, we reflect on the gaps that still need to be filled, what we can measure and what not, future subjective experiments, and new research trends that could help us to fully characterise Dialogue Understandability.

Dialogue Understandability: Why are we streaming movies with subtitles?

TL;DR

This work introduces Dialogue Understandability (DU) as a holistic measure of how viewers follow movie plots under streaming conditions, extending beyond traditional intelligibility to encompass cognitive effort and QoE. It proposes a working definition, a six-factor root-cause framework, and a mapping to QoE and the movie lifecycle, then evaluates existing objective tools through pilot experiments to assess their suitability for DU. The paper highlights gaps in measurement maturity, argues for adaptation and multi-source modeling (audio, video, subtitles, metadata), and discusses practical avenues such as dialogue enhancement and smart subtitling. Its contributions pave the way for targeted datasets and objective tools to predict and improve DU, with implications for accessibility, localization, and user experience in streaming services.

Abstract

Watching movies and TV shows with subtitles enabled is not simply down to audibility or speech intelligibility. A variety of evolving factors related to technological advances, cinema production and social behaviour challenge our perception and understanding. This study seeks to formalise and give context to these influential factors under a wider and novel term referred to as Dialogue Understandability. We propose a working definition for Dialogue Understandability being a listener's capacity to follow the story without undue cognitive effort or concentration being required that impacts their Quality of Experience (QoE). The paper identifies, describes and categorises the factors that influence Dialogue Understandability mapping them over the QoE framework, a media streaming lifecycle, and the stakeholders involved. We then explore available measurement tools in the literature and link them to the factors they could potentially be used for. The maturity and suitability of these tools is evaluated over a set of pilot experiments. Finally, we reflect on the gaps that still need to be filled, what we can measure and what not, future subjective experiments, and new research trends that could help us to fully characterise Dialogue Understandability.
Paper Structure (38 sections, 5 figures, 4 tables)

This paper contains 38 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Dialogue Understandability
  3. Working Definition
  4. Root causes
  5. Related Concepts
  6. Quality of Experience (QoE)
  7. Speech Quality
  8. Speech Intelligibility
  9. Loudness
  10. Speech Rate
  11. Speaker Identification
  12. Immersion
  13. Multimodal Interactions
  14. Influencing Factors
  15. Artistic/Production Style
  16. ...and 23 more sections

Figures (5)

  • Figure 1: The Dialogue Understandability Influencing Factors mapped to the QoE Influencing Factors .
  • Figure 2: Evolution of sound mixes from 1920 to 2020.
  • Figure 3: The Dialogue Understandability lifecycle and key stakeholders. The QoE Influencing Factors (top left) are mapped to stages of the lifecycle (creation, delivery, consumption) for each of the Dialogue Understandability Influencing Factors. Key stakeholders and systems at each stage are also illustrated.
  • Figure 4: Feature analysis for movie trailers. Slumdog Millionaire (Danny Boyle, Loveleen Tandan, 2008).
  • Figure 5: Measuring Dialogue Understandability. Feature sources and potential metrics to predict Dialogue Understandability.