Where is the multimodal goal post? On the Ability of Foundation Models to Recognize Contextually Important Moments

TL;DR

This work addresses the challenge of recognizing contextually important moments in temporally-ordered multimodal data by constructing MOMENTS, a football-focused dataset that aligns official highlights with full games. It systematically evaluates state-of-the-art foundation models across video, audio, and language modalities, finding performances near chance and limited cross-modal fusion gains, with visual cues driving important-moment detection and linguistic cues aiding non-important moments. Through modality-contribution analyses, the authors argue for modular, dynamic fusion architectures that can adapt to sample-level heterogeneity, and they release MOMENTS to spur progress in multimodal event understanding. Overall, the study highlights a substantial gap between current multimodal models and human-level performance in long-span, context-rich narratives and provides practical directions for improving cross-modal integration in foundation models.

Abstract

Foundation models are used for many real-world applications involving language generation from temporally-ordered multimodal events. In this work, we study the ability of models to identify the most important sub-events in a video, which is a fundamental prerequisite for narrating or summarizing multimodal events. Specifically, we focus on football games and evaluate models on their ability to distinguish between important and non-important sub-events in a game. To this end, we construct a new dataset by leveraging human preferences for importance implicit in football game highlight reels, without any additional annotation costs. Using our dataset, which we will publicly release to the community, we compare several state-of-the-art multimodal models and show that they are not far from chance level performance. Analyses of models beyond standard evaluation metrics reveal their tendency to rely on a single dominant modality and their ineffectiveness in synthesizing necessary information from multiple sources. Our findings underline the importance of modular architectures that can handle sample-level heterogeneity in multimodal data and the need for complementary training procedures that can maximize cross-modal synergy.

Figures (11)

• Figure 1: The moment on the top, showing a shot-on-target, was featured in the highlights of this football game. In contrast, the moment on the bottom, showing a corner kick, was not. A logistic regression classifier trained on the transcriptions of commentaries identified the text in blue as driving the prediction toward the 'important moment' class, while the text in orange drove the prediction toward the 'non-important moment'. The confidence of a foundation model, Qwen-2.5-Omni, in correctly identifying this 'important moment' is highest under vision-only input (V=3.81, L=-0.18, LV=-0.93), whereas for the 'non-important moment' multimodal information (language in the commentary complementing the visual) results in the highest model confidence (V=0.5, L=0.87, LV=1.34). More examples from our dataset are provided in Appendix \ref{['sec:appendix_rq2']} Figure \ref{['fig:moments_examples_supplementary']}.
• Figure 2: Among the 127 <$\text{H}$,$\text{G}$> pairs we process, more than half of the frames in 100 $\text{H}$ videos are accurately localized in their corresponding $\text{G}$ videos after step 2.
• Figure 3: The duration distributions of non-important moments (NIM) reflects the distributions of important ones (IM), with the average duration ($\bar{t}$) of the audio modality extending beyond the average of corresponding video frames for both classes.
• Figure 4: MCC and accuracy for all models under various combinations of modalities. We also compute uncertainties (line indicators on top of each bar) representing the 95% confidence interval, estimated via bootstrap resampling bootstrap_resampling. $\diamond$ denotes the highest scoring model under each modality combination and $\smallstar$ denotes the model with highest overall score.
• Figure 5: Contribution scores of the three modalities for the Qwen2.5-Omni-7B model separated for important (IM) and non-important (NIM) moments.