Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation

TL;DR

This work tackles the problem of missing modalities in pretrained multimodal models by introducing SURE, a framework that performs latent-space reconstruction and provides uncertainty estimates for both reconstructed inputs and downstream predictions. SURE integrates simple reconstruction modules after latent projections and employs a distribution-free Pearson Correlation-based loss, along with an error-propagation mechanism, to quantify how missing data impacts outputs. The approach is architecture-agnostic and demonstrated to achieve state-of-the-art results across sentiment analysis, book genre classification, and action recognition under incomplete data, while delivering calibrated uncertainty that can inform decision making. The practical impact lies in robust, uncertainty-aware predictions in safety-critical settings like healthcare and autonomous systems, where deferring uncertain decisions can improve reliability.

Abstract

Multimodal learning has demonstrated incredible successes by integrating diverse data sources, yet it often relies on the availability of all modalities - an assumption that rarely holds in real-world applications. Pretrained multimodal models, while effective, struggle when confronted with small-scale and incomplete datasets (i.e., missing modalities), limiting their practical applicability. Previous studies on reconstructing missing modalities have overlooked the reconstruction's potential unreliability, which could compromise the quality of the final outputs. We present SURE (Scalable Uncertainty and Reconstruction Estimation), a novel framework that extends the capabilities of pretrained multimodal models by introducing latent space reconstruction and uncertainty estimation for both reconstructed modalities and downstream tasks. Our method is architecture-agnostic, reconstructs missing modalities, and delivers reliable uncertainty estimates, improving both interpretability and performance. SURE introduces a unique Pearson Correlation-based loss and applies statistical error propagation in deep networks for the first time, allowing precise quantification of uncertainties from missing data and model predictions. Extensive experiments across tasks such as sentiment analysis, genre classification, and action recognition show that SURE consistently achieves state-of-the-art performance, ensuring robust predictions even in the presence of incomplete data.

Figures (12)

• Figure 1: Comparison of pretrained and vanilla MMML framework on CMU-MOSI dataset.
• Figure 2: SURE reconstructs missing modalities for final predictions, reporting average errors and uncertainties for both reconstruction (R.) and output (O.).
• Figure 3: Overview of SURE. Reconstruction modules $r^i(.)$’ are inserted after latent projections $f^i(.)$ and before fusion layers $\omega(.)$ in pretrained multimodal frameworks, reconstructing missing modalities with uncertainties. Reconstructed outputs replace missing inputs while propagating uncertainty, and the final classifier estimates output uncertainty from the model’s inherent variability. The notation $\widetilde{\bullet}$ distinguishes SURE trainable modules' estimates from inputs, outputs, and frozen module's intermediate variables.
• Figure 4: Correlation of estimated uncertainty with prediction error on UTD-MHAD dataset.
• Figure 5: Relationship between estimated output uncertainty and output error on UTD-MHAD test dataset.

Theorems & Definitions (3)

• Theorem 2.1
• proof
• Proposition 2.2