LLM-Integrated Bayesian State Space Models for Multimodal Time-Series Forecasting

TL;DR

This work addresses the challenge of multimodal time-series forecasting by unifying a probabilistic latent state-space model with a pretrained large language model (LLM) to jointly predict numeric values and generate textual forecasts. The LBS framework grounds both modalities in a shared latent trajectory, using a text-conditioned posterior via an LLM and state-conditioned text generation, trained with a multimodal evidence lower bound (ELBO) that includes value, text, and KL terms. Key innovations include text compression with summary tokens for posterior estimation, single shared LLM for encoding/decoding, and stateful training to manage computation, yielding principled uncertainty estimates and flexible horizons. On TimeTextCorpus, LBS achieves a $13.20\%$ improvement over prior state-of-the-art, provides human-readable forecast summaries, and reveals interpretable latent dynamics, suggesting strong potential for robust, interpretable multimodal temporal reasoning in climate, healthcare, and beyond.

Abstract

Forecasting in the real world requires integrating structured time-series data with unstructured textual information, but existing methods are architecturally limited by fixed input/output horizons and are unable to model or quantify uncertainty. We address this challenge by introducing LLM-integrated Bayesian State space models (LBS), a novel probabilistic framework for multimodal temporal forecasting. At a high level, LBS consists of two components: (1) a state space model (SSM) backbone that captures the temporal dynamics of latent states from which both numerical and textual observations are generated and (2) a pretrained large language model (LLM) that is adapted to encode textual inputs for posterior state estimation and decode textual forecasts consistent with the latent trajectory. This design enables flexible lookback and forecast windows, principled uncertainty quantification, and improved temporal generalization thanks to the well-suited inductive bias of SSMs toward modeling dynamical systems. Experiments on the TextTimeCorpus benchmark demonstrate that LBS improves the previous state-of-the-art by 13.20% while providing human-readable summaries of each forecast. Our work is the first to unify LLMs and SSMs for joint numerical and textual prediction, offering a novel foundation for multimodal temporal reasoning.

Figures (8)

• Figure 1: An illustration of the LBS architecture under for a single-step forecasting scenario. Bottom: To enable variational inference with text, the LLM is tuned to summarize the context into a set of summarization tokens, which are then used together with the target value to obtain the posterior state distribution. Top: Conditioned on the state forecast, the shared LLM is trained to generate its corresponding text. When using instruction-tuned LLMs, both steps are accompanied by prompt templates in order to preserve its capabilities.
• Figure 2: The latent dynamical model assumed by LBS. The SSM serves as a backbone modeling the temporal dynamics of latent states ${\bm{x}}_t$, from which multimodal data ${\bm{y}}_t$ and $\bm{{\mathcal{D}}}_t$ are generated.
• Figure 3: Illustration of a single forward pass through LBS.
• Figure 4: Test RMSE results from TTC-Climate (left) and TTC-Medical (right).
• Figure 5: Example text comparison generated by LBS vs. ground truth text from TTC-Climate. LBS is able to textually forecast key characteristics by contextualizing the LLM on the latent states.