Attend to what I say: Highlighting relevant content on slides

TL;DR

This work tackles aligning a speaker's narration with slide content to reduce cognitive load during presentations by automatically identifying and highlighting slide regions informed by OCR and ASR outputs. It introduces a multi-modal framework that combines string and semantic alignment with LLM-based refinement to map transcript lines to slide regions and then highlight them, using bounding boxes, shading, or magnification. Evaluation centers on $S_c$, $S_m$, and F1 across methods and thresholds, revealing a trade-off: simple fuzzy matching yields high $S_c$ but high $S_m$, while embedding- and LLM-based approaches offer more balanced performance, especially when ASR outputs are corrected with OCR guidance. A conference-presentation dataset (NeurIPS 2022, ICML 2023) with 150 slides and over an hour of content supports the analysis, and a user study favors bounding-box highlighting, indicating practical benefits for live or recorded presentations and future integration with visual-language models for real-time alignment.

Abstract

Imagine sitting in a presentation, trying to follow the speaker while simultaneously scanning the slides for relevant information. While the entire slide is visible, identifying the relevant regions can be challenging. As you focus on one part of the slide, the speaker moves on to a new sentence, leaving you scrambling to catch up visually. This constant back-and-forth creates a disconnect between what is being said and the most important visual elements, making it hard to absorb key details, especially in fast-paced or content-heavy presentations such as conference talks. This requires an understanding of slides, including text, graphics, and layout. We introduce a method that automatically identifies and highlights the most relevant slide regions based on the speaker's narrative. By analyzing spoken content and matching it with textual or graphical elements in the slides, our approach ensures better synchronization between what listeners hear and what they need to attend to. We explore different ways of solving this problem and assess their success and failure cases. Analyzing multimedia documents is emerging as a key requirement for seamless understanding of content-rich videos, such as educational videos and conference talks, by reducing cognitive strain and improving comprehension. Code and dataset are available at: https://github.com/meghamariamkm2002/Slide_Highlight

Figures (14)

• Figure 1: This illustration highlights the challenge audiences face during presentations in simultaneously locating relevant content on slides while paying attention to the speaker. By synchronizing spoken information with corresponding visual elements, cognitive load is reduced, engagement is improved, and comprehension and retention of key insights are enhanced.
• Figure 2: This figure illustrates the pipeline of our method. The Alignment Module takes ocr-extracted text and asr-generated transcripts as inputs to establish correspondences between spoken and visual content. These aligned results are then passed to the Highlighting Module, from which a suitable visualization can be chosen to highlight the relevant slide content.
• Figure 3: This figure showcases a set of presentation slides from our dataset. The dataset consists of slides with varying layouts, color schemes, and content structures, reflecting the diversity of real-world academic and professional presentations.
• Figure 4: Statistics for Audio Segment Durations, Word Count in asr and ocr
• Figure 5: The distribution of presentation slides based on the duration of their corresponding audio segments.