Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification

TL;DR

This work addresses the discarding-imputation dilemma in incomplete multimodal learning by introducing DyMo, an inference-time framework that dynamically selects and fuses recovered modalities to maximize task-relevant information. Central to DyMo is MTIR, a reward informed by a bound that ties changes in test-time cross-entropy loss to information gain, guiding per-sample modality inclusion, augmented by intra-class similarity calibration for robustness. The method features a flexible multimodal architecture, an iterative selection algorithm, and a training strategy with incomplete-modality simulation and contrastive learning, achieving state-of-the-art results across five diverse datasets and various missing-data scenarios. DyMo improves practical deployability of multimodal models by effectively leveraging high-quality recovered information while suppressing unreliable or misaligned reconstructions, with strong empirical and theoretical foundations for inference-time decisions.

Abstract

Multimodal deep learning (MDL) has achieved remarkable success across various domains, yet its practical deployment is often hindered by incomplete multimodal data. Existing incomplete MDL methods either discard missing modalities, risking the loss of valuable task-relevant information, or recover them, potentially introducing irrelevant noise, leading to the discarding-imputation dilemma. To address this dilemma, in this paper, we propose DyMo, a new inference-time dynamic modality selection framework that adaptively identifies and integrates reliable recovered modalities, fully exploring task-relevant information beyond the conventional discard-or-impute paradigm. Central to DyMo is a novel selection algorithm that maximizes multimodal task-relevant information for each test sample. Since direct estimation of such information at test time is intractable due to the unknown data distribution, we theoretically establish a connection between information and the task loss, which we compute at inference time as a tractable proxy. Building on this, a novel principled reward function is proposed to guide modality selection. In addition, we design a flexible multimodal network architecture compatible with arbitrary modality combinations, alongside a tailored training strategy for robust representation learning. Extensive experiments on diverse natural and medical image datasets show that DyMo significantly outperforms state-of-the-art incomplete/dynamic MDL methods across various missing-data scenarios. Our code is available at https://github.com//siyi-wind/DyMo.

Figures (11)

• Figure 1: (a-b) Evidence of the discarding-imputation dilemma: (a-1) vs. (a-2) recovery-free methods (e.g., ModDrop neverova2015moddrop) learn less discriminative features because they ignore highly task-relevant missing modalities {M,T}; (b) recovery-based methods (e.g., MoPoE suttergeneralized) generate unreliable reconstructions, e.g., low-fidelity (orange) or misaligned (yellow) modalities. (c) Our DyMo, which addresses the dilemma by dynamically fusing task-relevant recovered modalities, improving accuracy by 5.67% on PolyMNIST and 1.68% on MST (Tab. \ref{['tab:SOTA_missing']}).
• Figure 2: Multimodal network architecture $f$ for arbitrary modalities.
• Figure 3: Comparison of DyMo with static/dynamic multimodal fusion techniques on 6 multimodal classification tasks under various missing scenarios. DyMo$_{c}$ and DyMo$_e$ denote the use of cosine and squared Euclidean distances, respectively.
• Figure 4: (a) t-SNE visualization of DyMo$_c$ on MST with different modality inputs: (a-1) using only non-missing modalities; (a-2) integrating all recovered modalities without selection; (a-3) incorporating recovered modalities selected by DyMo$_c$. (b) PCA visualizations of two successful DyMo$_c$'s test cases on DVM: (b-1) a misprediction corrected by incorporating a recovered modality; (b-2) a correct prediction maintained by disregarding an unreliable recovered modality.
• Figure S1: (a) Performance of DyMo$_c$'s multimodal network on MST under various missing scenarios, evaluated with different numbers of sampled modality subsets ($A$). w/o A denotes training without our incomplete-modality simulation strategy. (b) Comparison between MTL and DyMo$_e$ on MST with {M,T} missing, under different modality inputs: (1) using only non-missing modalities; (2) integrating all recovered modalities without selection; (3) integrating only the recovered modalities selected by DyMo$_e$.