DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning

TL;DR

DynaStride addresses the challenge of generating coherent scene-level captions for instructional videos by introducing a four-stage pipeline that adaptively samples frames, generates multimodal chain-of-thought subcaptions, dynamically selects informative windows, and aggregates them into concise scene descriptions. The approach leverages MMCoT prompting with Qwen models and a dynamic stride algorithm to balance temporal context and redundancy, evaluated on YouCookII against strong baselines (GPT-4o, VLLaMA-3) with improvements in CIDEr and semantic similarity metrics. Key contributions include a practical frame-windowing strategy, an adaptive window selection mechanism, and an aggregation scheme that yields temporally coherent, instructional captions. The results highlight the potential of MMCoT-based multimodal reasoning for enhancing AI-generated instructional content and point to avenues for domain expansion and longer-range cross-modal modeling.

Abstract

Scene-level captioning in instructional videos can enhance learning by requiring an understanding of both visual cues and temporal structure. By aligning visual cues with textual guidance, this understanding supports procedural learning and multimodal reasoning, providing a richer context for skill acquisition. However, captions that fail to capture this structure may lack coherence and quality, which can create confusion and undermine the video's educational intent. To address this gap, we introduce DynaStride, a pipeline to generate coherent, scene-level captions without requiring manual scene segmentation. Using the YouCookII dataset's scene annotations, DynaStride performs adaptive frame sampling and multimodal windowing to capture key transitions within each scene. It then employs a multimodal chain-of-thought process to produce multiple action-object pairs, which are refined and fused using a dynamic stride window selection algorithm that adaptively balances temporal context and redundancy. The final scene-level caption integrates visual semantics and temporal reasoning in a single instructional caption. Empirical evaluations against strong baselines, including VLLaMA3 and GPT-4o, demonstrate consistent gains on both N-gram-based metrics (BLEU, METEOR) and semantic similarity measures (BERTScore, CLIPScore). Qualitative analyses further show that DynaStride produces captions that are more temporally coherent and informative, suggesting a promising direction for improving AI-powered instructional content generation.

Figures (1)

• Figure 1: Illustration of the multi-scene-to-text captioning pipeline depicting four stages: frame sampling, MMCoT, dynamic stride, and aggregation.