Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Qian Chen, Jinlan Fu, Changsong Li, See-Kiong Ng, Xipeng Qiu

TL;DR

This work introduces FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments and proposes an Omni-Modal Future Forecasting (OFF) training strategy, which enhances future forecasting and generalization.

Abstract

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

TL;DR

This work introduces FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments and proposes an Omni-Modal Future Forecasting (OFF) training strategy, which enhances future forecasting and generalization.

Abstract

Although Multimodal Large Language Models (MLLMs) demonstrate strong omni-modal perception, their ability to forecast future events from audio-visual cues remains largely unexplored, as existing benchmarks focus mainly on retrospective understanding. To bridge this gap, we introduce FutureOmni, the first benchmark designed to evaluate omni-modal future forecasting from audio-visual environments. The evaluated models are required to perform cross-modal causal and temporal reasoning, as well as effectively leverage internal knowledge to predict future events. FutureOmni is constructed via a scalable LLM-assisted, human-in-the-loop pipeline and contains 919 videos and 1,034 multiple-choice QA pairs across 8 primary domains. Evaluations on 13 omni-modal and 7 video-only models show that current systems struggle with audio-visual future prediction, particularly in speech-heavy scenarios, with the best accuracy of 64.8% achieved by Gemini 3 Flash. To mitigate this limitation, we curate a 7K-sample instruction-tuning dataset and propose an Omni-Modal Future Forecasting (OFF) training strategy. Evaluations on FutureOmni and popular audio-visual and video-only benchmarks demonstrate that OFF enhances future forecasting and generalization. We publicly release all code (https://github.com/OpenMOSS/FutureOmni) and datasets (https://huggingface.co/datasets/OpenMOSS-Team/FutureOmni).
Paper Structure (31 sections, 7 figures, 7 tables)

This paper contains 31 sections, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The FutureOmni Benchmark
  4. Audio Coordinated Video Selection
  5. Audio-Visual Temporal Localization and Calibration
  6. Audio-Visual QA Construction
  7. Dataset Statistics
  8. Experiments
  9. Settings
  10. Results on FutureOmni
  11. Breakdown Results
  12. Modality Ablation
  13. Error Analysis
  14. Omnimodal Future Forecasting: From Prediction to Generalization
  15. Main Experiment
  16. ...and 16 more sections

Figures (7)

  • Figure 1: An Example from FutureOmni illustrating the Omni-modal Future Prediction task. The green and pink arrows denote consequences induced by auditory and visual cues, respectively.
  • Figure 2: Overall scores on FutureOmni.
  • Figure 3: The pipeline of our FutureOmni.
  • Figure 4: (a) Hierarchical distribution of 8 primary video domains and 21 fine-grained sub-categories. Surv: Surveillance, Daily: Dailylife, Edu: Education, Emerg: Emergency. (b) Composition of audio modalities and Average video duration.
  • Figure 5: Error Distribution. I: Lack of Knowledge. II: Video Perception Error. III: Audio Perception Error. IV: Reasoning Failure.
  • ...and 2 more figures