Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Multimodal Learning via Entropy-Gated Contrastive Fusion

Leon Chlon, Maggie Chlon, MarcAntonio M. Awada

TL;DR

This work tackles robust, calibrated multimodal inference under missing modalities by introducing Adaptive Entropy-Gated Contrastive Fusion (AECF), a lightweight fusion layer that operates on frozen encoders. It combines three synergistic modules: a meta-adaptive entropy-regularised gate to prevent modality collapse, Contrastive Expert Calibration (CEC) to enforce monotone confidence across all modality subsets, and Adaptive Curriculum Masking (ACM) to adversarially probe dominant modalities during training. Theoretical guarantees include a worst-case subset regret bound and a PAC-like calibration bound, together with empirical gains on AV-MNIST and MS-COCO, where masking yields up to +18 pp mAP improvements and ECE reductions up to ~200% with minimal runtime overhead. By providing a drop-in fusion layer that preserves accuracy on full inputs while improving robustness and calibration under partial observations, AECF offers a practical path to reliable multimodal deployment with frozen backbones.

Abstract

Real-world multimodal systems routinely face missing-input scenarios, and in reality, robots lose audio in a factory or a clinical record omits lab tests at inference time. Standard fusion layers either preserve robustness or calibration but never both. We introduce Adaptive Entropy-Gated Contrastive Fusion (AECF), a single light-weight layer that (i) adapts its entropy coefficient per instance, (ii) enforces monotone calibration across all modality subsets, and (iii) drives a curriculum mask directly from training-time entropy. On AV-MNIST and MS-COCO, AECF improves masked-input mAP by +18 pp at a 50% drop rate while reducing ECE by up to 200%, yet adds 1% run-time. All back-bones remain frozen, making AECF an easy drop-in layer for robust, calibrated multimodal inference.

Robust Multimodal Learning via Entropy-Gated Contrastive Fusion

TL;DR

This work tackles robust, calibrated multimodal inference under missing modalities by introducing Adaptive Entropy-Gated Contrastive Fusion (AECF), a lightweight fusion layer that operates on frozen encoders. It combines three synergistic modules: a meta-adaptive entropy-regularised gate to prevent modality collapse, Contrastive Expert Calibration (CEC) to enforce monotone confidence across all modality subsets, and Adaptive Curriculum Masking (ACM) to adversarially probe dominant modalities during training. Theoretical guarantees include a worst-case subset regret bound and a PAC-like calibration bound, together with empirical gains on AV-MNIST and MS-COCO, where masking yields up to +18 pp mAP improvements and ECE reductions up to ~200% with minimal runtime overhead. By providing a drop-in fusion layer that preserves accuracy on full inputs while improving robustness and calibration under partial observations, AECF offers a practical path to reliable multimodal deployment with frozen backbones.

Abstract

Real-world multimodal systems routinely face missing-input scenarios, and in reality, robots lose audio in a factory or a clinical record omits lab tests at inference time. Standard fusion layers either preserve robustness or calibration but never both. We introduce Adaptive Entropy-Gated Contrastive Fusion (AECF), a single light-weight layer that (i) adapts its entropy coefficient per instance, (ii) enforces monotone calibration across all modality subsets, and (iii) drives a curriculum mask directly from training-time entropy. On AV-MNIST and MS-COCO, AECF improves masked-input mAP by +18 pp at a 50% drop rate while reducing ECE by up to 200%, yet adds 1% run-time. All back-bones remain frozen, making AECF an easy drop-in layer for robust, calibrated multimodal inference.

Paper Structure

This paper contains 53 sections, 4 theorems, 16 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Deep MoE and multimodal robustness.
  5. Calibration and uncertainty under missing modalities.
  6. Positioning of AECF.
  7. AECF
  8. Problem setting and notation
  9. Composite objective.
  10. Standing assumption.
  11. Module 1: Meta-adaptive entropy-regularised gating
  12. Entropy regularisation to avoid collapse.
  13. Adaptive coefficient.
  14. Optimisation view.
  15. Module 2: Contrastive Expert calibration
  16. ...and 38 more sections

Key Result

Theorem 1

Let $\widehat{g}_S$ be the calibrated scores after $T$ CEC updates with learning rate $\eta\le\frac{1}{L}$. With probability at least $1-\delta$ over a validation set of size $N$, where $g_S^\star$ is the lattice-isotonic optimum

Figures (2)

  • Figure 1: AECF pipeline. A two--layer gate mixes frozen encoder features. During training the gate entropy $H(p)$ is penalised (coefficient $\lambda_t$ follows the epoch-dependent schedule in §3.3) while inputs are randomly masked with probability $\pi_t$ that linearly ramps to $\pi_{\max}$. Both schedules are deterministic once training starts; at test time no masking is applied.
  • Figure 2: Per-sample gate entropy $H(p)$ versus model confidence ($\max_k\sigma(\text{logit}_k)$). Only the full AECF model (a) shows the expected monotone relationship, corroborating the theory of §\ref{['sec:cec_monotone']}.

Theorems & Definitions (8)

  • Theorem 1
  • Theorem 2
  • proof
  • Theorem 3
  • proof
  • Lemma 4
  • proof
  • proof : Proof of Theorem \ref{['thm:pac_ece']}