Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-Forecast: A Multimodal Approach to Temporal Event Forecasting with Large Language Models

Haoxuan Li, Zhengmao Yang, Yunshan Ma, Yi Bin, Yang Yang, Tat-Seng Chua

TL;DR

This work addresses the gap in multimodal temporal event forecasting by introducing MM-Forecast, a framework that identifies two image functions—highlighting and complementary—and translates them into verbal prompts for LLM-based forecasting. The approach supports both in-context learning and retrieval-augmented generation, and is evaluated on a newly constructed MidEast-TE-mm dataset enriched with images. Empirical results show that incorporating image-function descriptions improves forecasting accuracy, with ablations confirming the value of combining highlighting and complementary information and demonstrating high image-function identification accuracy. The findings highlight the potential of multimodal LLMs for forecasting and point to future work on multi-image relationships, credibility, and end-to-end integration.

Abstract

We study an emerging and intriguing problem of multimodal temporal event forecasting with large language models. Compared to using text or graph modalities, the investigation of utilizing images for temporal event forecasting has not been fully explored, especially in the era of large language models (LLMs). To bridge this gap, we are particularly interested in two key questions of: 1) why images will help in temporal event forecasting, and 2) how to integrate images into the LLM-based forecasting framework. To answer these research questions, we propose to identify two essential functions that images play in the scenario of temporal event forecasting, i.e., highlighting and complementary. Then, we develop a novel framework, named MM-Forecast. It employs an Image Function Identification module to recognize these functions as verbal descriptions using multimodal large language models (MLLMs), and subsequently incorporates these function descriptions into LLM-based forecasting models. To evaluate our approach, we construct a new multimodal dataset, MidEast-TE-mm, by extending an existing event dataset MidEast-TE-mini with images. Empirical studies demonstrate that our MM-Forecast can correctly identify the image functions, and further more, incorporating these verbal function descriptions significantly improves the forecasting performance. The dataset, code, and prompts are available at https://github.com/LuminosityX/MM-Forecast.

MM-Forecast: A Multimodal Approach to Temporal Event Forecasting with Large Language Models

TL;DR

This work addresses the gap in multimodal temporal event forecasting by introducing MM-Forecast, a framework that identifies two image functions—highlighting and complementary—and translates them into verbal prompts for LLM-based forecasting. The approach supports both in-context learning and retrieval-augmented generation, and is evaluated on a newly constructed MidEast-TE-mm dataset enriched with images. Empirical results show that incorporating image-function descriptions improves forecasting accuracy, with ablations confirming the value of combining highlighting and complementary information and demonstrating high image-function identification accuracy. The findings highlight the potential of multimodal LLMs for forecasting and point to future work on multi-image relationships, credibility, and end-to-end integration.

Abstract

We study an emerging and intriguing problem of multimodal temporal event forecasting with large language models. Compared to using text or graph modalities, the investigation of utilizing images for temporal event forecasting has not been fully explored, especially in the era of large language models (LLMs). To bridge this gap, we are particularly interested in two key questions of: 1) why images will help in temporal event forecasting, and 2) how to integrate images into the LLM-based forecasting framework. To answer these research questions, we propose to identify two essential functions that images play in the scenario of temporal event forecasting, i.e., highlighting and complementary. Then, we develop a novel framework, named MM-Forecast. It employs an Image Function Identification module to recognize these functions as verbal descriptions using multimodal large language models (MLLMs), and subsequently incorporates these function descriptions into LLM-based forecasting models. To evaluate our approach, we construct a new multimodal dataset, MidEast-TE-mm, by extending an existing event dataset MidEast-TE-mini with images. Empirical studies demonstrate that our MM-Forecast can correctly identify the image functions, and further more, incorporating these verbal function descriptions significantly improves the forecasting performance. The dataset, code, and prompts are available at https://github.com/LuminosityX/MM-Forecast.
Paper Structure (27 sections, 8 figures, 7 tables)

This paper contains 27 sections, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Temporal Event Forecasting
  4. LLMs for Event Analysis
  5. Our Approach: MM-Forecast
  6. Problem Formulation
  7. Image Function Identification
  8. Forecasting Framework
  9. In-context Learning (ICL)
  10. Retrieval Augmented Generation (RAG)
  11. Experiments
  12. Experimental Settings
  13. Dataset
  14. Compared Methods
  15. Implementation Details
  16. ...and 12 more sections

Figures (8)

  • Figure 1: Illustration of our motivation about why images will help in temporal event forecasting. We identify two essential functions of images, i.e., highlighting and complementary. By offering auxiliary highlighting or complementary information, images enhance the understanding of temporal events, thus boosting the forecasting performance.
  • Figure 2: The schematic overview of MM-Forecast. By consuming historical events in either format of unstructured or structured input (left), our image function identification module (middle) recognizes the image functions as verbal descriptions, which are then feed into LLM-based forecasting model (right). Our framework is versatile to handle both structured and unstructured events, meanwhile, it is compatible to popular LLM components for event forecasting, i.e., ICL and RAG.
  • Figure 3: Ablation study of each type of image functions.
  • Figure 4: Case study: two examples that when considering highlighting and complementary functions of images, our method yields better forecasting results compared with the baselines.
  • Figure 5: The case study of highlighting function of image.
  • ...and 3 more figures