Useful but Distracting: Keyword Highlights and Time-Synchronization in Captions for Language Learning

TL;DR

This work investigates enhanced captions designed to support language learning by highlighting keywords and synchronizing them with audio. Through a user-centered design process, the authors implement four caption variants and evaluate them via a survey, focus group, and a 49-participant online study using clips from Marriage Story. Results show that time-synchronized keyword highlights and keyword-highlight captions improve learning-oriented perceptions but are perceived as distracting for everyday viewing, with standard captions remaining the most favorable for comprehension and entertainment. The study highlights the potential of keyword-centric captions to support learning within real-world viewing, while underscoring the need to optimize designs to minimize distraction and preserve enjoyment. Practical implications include using conservative keyword selection, context-preserving full captions, and potentially limiting enhancements to learning-focused viewing contexts or curricula-aligned deployments.

Abstract

Captions provide language learners with a scaffold for comprehension and vocabulary acquisition. Past work has proposed several enhancements such as keyword highlights for increased learning gains. However, little is known about learners' experience with enhanced captions, although this is critical for adoption in everyday life. We conducted a survey and focus group to elicit learner preferences and requirements and implemented a processing pipeline for enhanced captions with keyword highlights, time-synchronized keyword highlights, and keyword captions. A subsequent online study (n = 49) showed that time-synchronized keyword highlights were the preferred design for learning but were perceived as too distracting to replace standard captions in everyday viewing scenarios. We conclude that keyword highlights and time-synchronization are suitable for integrating learning into an entertaining everyday-life activity, but the design should be optimized to provide a more seamless experience.

Figures (3)

• Figure 1: Processing pipeline for subtitle files
• Figure 2: UEQ-S scores of the four caption types compared in our user study.
• Figure 3: Ranking of the caption types for the purposes comprehension, entertainment, and learning