AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learning

TL;DR

AdaFuse tackles the challenge that fixed multimodal fusion may include uninformative modalities for individual patients. It introduces a reinforcement-learning framework that performs sequential modality selection, encoding each modality to a shared $d=32$ representation and using a policy to decide which subset of modalities to use, with early termination when sufficient information is obtained. The method achieves state-of-the-art AUC on the NLST dataset ($0.762$) compared with fixed fusion and other adaptive baselines, while reducing FLOPs by skipping uninformative modalities. External validation on VLSP shows robustness to distribution shift, illustrating that adaptive fusion can yield personalized, computation-efficient diagnostic pipelines in medical imaging.

Abstract

Multimodal fusion has emerged as a promising paradigm for disease diagnosis and prognosis, integrating complementary information from heterogeneous data sources such as medical images, clinical records, and radiology reports. However, existing fusion methods process all available modalities through the network, either treating them equally or learning to assign different contribution weights, leaving a fundamental question unaddressed: for a given patient, should certain modalities be used at all? We present AdaFuse, an adaptive multimodal fusion framework that leverages reinforcement learning (RL) to learn patient-specific modality selection and fusion strategies for lung cancer risk prediction. AdaFuse formulates multimodal fusion as a sequential decision process, where the policy network iteratively decides whether to incorporate an additional modality or proceed to prediction based on the information already acquired. This sequential formulation enables the model to condition each selection on previously observed modalities and terminate early when sufficient information is available, rather than committing to a fixed subset upfront. We evaluate AdaFuse on the National Lung Screening Trial (NLST) dataset. Experimental results demonstrate that AdaFuse achieves the highest AUC (0.762) compared to the best single-modality baseline (0.732), the best fixed fusion strategy (0.759), and adaptive baselines including DynMM (0.754) and MoE (0.742), while using fewer FLOPs than all triple-modality methods. Our work demonstrates the potential of reinforcement learning for personalized multimodal fusion in medical imaging, representing a shift from uniform fusion strategies toward adaptive diagnostic pipelines that learn when to consult additional modalities and when existing information suffices for accurate prediction.

Figures (5)

• Figure 1: Comparison of multimodal fusion paradigms. Existing multimodal fusion methods (left two) process all modalities uniformly: operation-based fusion applies fixed combination rules, while Mixture-of-Experts learns soft weights but still requires all inputs. AdaFuse (right) makes sequential discrete decisions to select patient-specific modality subsets, with the flexibility to entirely exclude uninformative modalities from computation.
• Figure 2: Overview of the AdaFuse framework.Data preprocessing extracts features from three modalities (CT images, clinical variables, and text reports) using pretrained encoders; details are provided in Section \ref{['sec:dataset']}. The AdaFuse framework consists of three components: (1) Modality encoders project each input feature to a shared 32-dimensional representation, with a binary mask $m\in \{0, 1\}^3$ tracking selected modalities.(2) State encoder concatenates the masked features with the selection mask and maps them to a 64-dimensional state vector that captures the current selection status. (3) Policy heads make sequential decisions: Step 1 selects the primary modality from $\{A, B, C\}$ ; Step 2 decides whether to add a second modality or proceed to prediction; Step 3 optionally incorporates the third modality and selects a fusion strategy from $\{Concat, Mean, Tensor\}$. After each selection, the corresponding mask entry is updated from 0 to 1, and the state encoder recomputes the state representation for the next decision. The selected modality combination is passed to the corresponding pretrained classifier among 15 fusion classifiers covering all valid modality-fusion combinations.
• Figure 3: Prediction correlation across baseline models. Models containing CT features exhibit high mutual correlation, while text-only predictions show near-zero correlation with others.
• Figure 4: Ablation study on training configurations. From left to right: (1) freezing classifiers while training modality encoders; (2) freezing modality encoders while training classifiers; (3) training both components; (4) freezing both components. The flame icon indicates which components receive gradients during RL training. Quantitative results are provided in Table \ref{['tab:ablation_training']}.
• Figure 5: AdaFuse policy selection distribution and modality skip rates on the test set. Left: Distribution of modality-fusion combinations selected by the learned policy, where $N$ denotes the number of patients. The policy most frequently selects ABC-concat ($N=288$, 62.3%), followed by AB-tensor ($N=54$, 11.7%) and clinical-only ($N=37$, 8.0%). Right: Frequency of skipping each modality. The text modality (C) is skipped for 143 patients (31.0%), while CT (A) and clinical variables (B) are each skipped for only 12 patients (2.6%). This confirms that the policy learns to filter out the less informative text modality while consistently relying on imaging and clinical data.