Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Context-specific Credibility-aware Multimodal Fusion with Conditional Probabilistic Circuits

Pranuthi Tenali, Sahil Sidheekh, Saurabh Mathur, Erik Blasch, Kristian Kersting, Sriraam Natarajan

Abstract

Multimodal fusion requires integrating information from multiple sources that may conflict depending on context. Existing fusion approaches typically rely on static assumptions about source reliability, limiting their ability to resolve conflicts when a modality becomes unreliable due to situational factors such as sensor degradation or class-specific corruption. We introduce C$^2$MF, a context-specfic credibility-aware multimodal fusion framework that models per-instance source reliability using a Conditional Probabilistic Circuit (CPC). We formalize instance-level reliability through Context-Specific Information Credibility (CSIC), a KL-divergence-based measure computed exactly from the CPC. CSIC generalizes conventional static credibility estimates as a special case, enabling principled and adaptive reliability assessment. To evaluate robustness under cross-modal conflicts, we propose the Conflict benchmark, in which class-specific corruptions deliberately induce discrepancies between different modalities. Experimental results show that C$^2$MF improves predictive accuracy by up to 29% over static-reliability baselines in high-noise settings, while preserving the interpretability advantages of probabilistic circuit-based fusion.

Context-specific Credibility-aware Multimodal Fusion with Conditional Probabilistic Circuits

Abstract

Multimodal fusion requires integrating information from multiple sources that may conflict depending on context. Existing fusion approaches typically rely on static assumptions about source reliability, limiting their ability to resolve conflicts when a modality becomes unreliable due to situational factors such as sensor degradation or class-specific corruption. We introduce CMF, a context-specfic credibility-aware multimodal fusion framework that models per-instance source reliability using a Conditional Probabilistic Circuit (CPC). We formalize instance-level reliability through Context-Specific Information Credibility (CSIC), a KL-divergence-based measure computed exactly from the CPC. CSIC generalizes conventional static credibility estimates as a special case, enabling principled and adaptive reliability assessment. To evaluate robustness under cross-modal conflicts, we propose the Conflict benchmark, in which class-specific corruptions deliberately induce discrepancies between different modalities. Experimental results show that CMF improves predictive accuracy by up to 29% over static-reliability baselines in high-noise settings, while preserving the interpretability advantages of probabilistic circuit-based fusion.

Paper Structure

This paper contains 22 sections, 8 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Multimodal fusion and reliability
  4. Probabilistic Circuits as fusion functions
  5. Context-Specific Credibility-Aware Multimodal Fusion
  6. Unimodal Encoders and Predictors.
  7. Joint Context Embedding.
  8. Fusion via Conditional Probabilistic Circuits.
  9. Context-specific Information credibility
  10. Learning
  11. Experimental Setup
  12. The Conflict Benchmark
  13. Conflict-AV-MNIST
  14. Conflict-NYUD
  15. Baselines
  16. ...and 7 more sections

Figures (2)

  • Figure 1: Credibility-Aware Fusion using Latent Context. Unlike static fusion methods, the C$^2$MF framework extracts a joint neural embedding that serves as a context for a Conditional Probabilistic Circuit. The circuit dynamically evaluates the credibility of each source based on its position in the latent space (i.e., identifying when one modality is unreliable), ensuring that the final prediction is dominated by the most reliable information source for that specific instance.
  • Figure 2: Mean test RMIS under decoupled training across varying test conflict levels $\lambda_\text{test}$. Top row: overall RMIS across all methods. Bottom row: RMIS broken down by corrupted modality for CWM vs. C$^2$WM, revealing whether credibility estimates adapt to the identity of the corrupted source or remain biased toward a single modality.