How does longer temporal context enhance multimodal narrative video processing in the brain?

TL;DR

This work tackles how longer temporal context and narrative prompting shape brain–model alignment during naturalistic movie viewing. It compares multimodal large language models (MLLMs) Qwen-2.5-Omni and DATE with unimodal baselines across sliding windows of $3$, $6$, $9$, and $12$ s, using voxel-wise encoding on Movie10 fMRI data and four narrative tasks. The key findings are that longer context significantly boosts brain predictivity for MLLMs, with a regional gradient from short timescales in perceptual areas to long timescales in PCC/dmPFC, and that narrative prompts yield task- and ROI-specific patterns of alignment; model layer depth mirrors cortical processing hierarchies. Overall, the results position long-form narrative movies as a principled testbed for biologically grounded long-context representations in AI, and demonstrate that narrative prompting provides a functional probe for brain-aligned multimodal representations.

Abstract

Understanding how humans and artificial intelligence systems process complex narrative videos is a fundamental challenge at the intersection of neuroscience and machine learning. This study investigates how the temporal context length of video clips (3--12 s clips) and the narrative-task prompting shape brain-model alignment during naturalistic movie watching. Using fMRI recordings from participants viewing full-length movies, we examine how brain regions sensitive to narrative context dynamically represent information over varying timescales and how these neural patterns align with model-derived features. We find that increasing clip duration substantially improves brain alignment for multimodal large language models (MLLMs), whereas unimodal video models show little to no gain. Further, shorter temporal windows align with perceptual and early language regions, while longer windows preferentially align higher-order integrative regions, mirrored by a layer-to-cortex hierarchy in MLLMs. Finally, narrative-task prompts (multi-scene summary, narrative summary, character motivation, and event boundary detection) elicit task-specific, region-dependent brain alignment patterns and context-dependent shifts in clip-level tuning in higher-order regions. Together, our results position long-form narrative movies as a principled testbed for probing biologically relevant temporal integration and interpretable representations in long-context MLLMs.

Figures (15)

• Figure 1: Leveraging temporal video context of different durations ($X_{\text{windows}}$) with unimodal and multimodal models for brain encoding with a diverse set of instructions (prompts). We experiment with 4 narrative video understanding tasks: character motivation, event boundary detection, multi-scene summarization, narrative summarization.
• Figure 2: Average normalized brain alignment as a function of temporal window length (3 to 12s) for MLLMs, and unimodal video baselines. MLLMs show increasing alignment with longer windows, while unimodal video models remain approximately constant. Error bars denote variability across subjects (mean $\pm$ standard error of the mean).
• Figure 3: Each voxel is color-coded with the video duration (window length) that led to the highest normalized brain alignment with the Qwen-2.5-Omni model. The color bar highlights color codes for each window length. (Left) The voxels are projected onto the flattened cortical surface of the 'fsaverage' subject. (Right): Percentage of best predicted voxels whose brain encoding performance is higher corresponding to each window length within language-selective regions, and visual regions. Results for DATE are in Appendix \ref{['app:window-wise-alignment-date-timesformer']}.
• Figure 4: Layer-wise alignment results for Qwen-2.5-Omni: Each voxel is color coded with the MLLM layer number (out of 36) that led to the highest normalized brain alignment. The color bar highlights color codes for each layer. The voxels are projected onto the flattened cortical surface of the 'fsaverage' subject. Results for DATE are in Appendix \ref{['app:layer-wise-date-timesformer']}.
• Figure 5: (Left) Each voxel is color-coded with the instruction that led to the highest normalized brain alignment for Qwen-2.5-Omni model. The color bar highlights color codes for each instruction. The voxels are projected onto the flattened cortical surface of the 'fsaverage' subject. (Right) Percentage of voxels in each ROI corresponding to each instruction. Results for DATE are in Appendix \ref{['app:task-wise-alignment-date-timesformer']}.