Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes

Valerio Guarrasi, Paolo Soda

TL;DR

This work addresses prognosis of COVID-19 by fusing chest X-ray and clinical data with a novel Pareto multi-objective framework that selects and combines unimodal models across modalities. It introduces a joint–late fusion scheme that concatenates unimodal outputs and trains an end-to-end classifier, guided by both accuracy and diversity objectives. The approach achieves state-of-the-art results on the AIforCOVID dataset and shows robustness to external validation, while enabling XAI-based interpretation that reveals modality hierarchy and intra-modality feature importance. The findings demonstrate how automated model- and modality-selection can yield interpretable, high-performance multimodal predictions with practical implications for clinical risk stratification.

Abstract

The COVID-19 pandemic has caused millions of cases and deaths and the AI-related scientific community, after being involved with detecting COVID-19 signs in medical images, has been now directing the efforts towards the development of methods that can predict the progression of the disease. This task is multimodal by its very nature and, recently, baseline results achieved on the publicly available AIforCOVID dataset have shown that chest X-ray scans and clinical information are useful to identify patients at risk of severe outcomes. While deep learning has shown superior performance in several medical fields, in most of the cases it considers unimodal data only. In this respect, when, which and how to fuse the different modalities is an open challenge in multimodal deep learning. To cope with these three questions here we present a novel approach optimizing the setup of a multimodal end-to-end model. It exploits Pareto multi-objective optimization working with a performance metric and the diversity score of multiple candidate unimodal neural networks to be fused. We test our method on the AIforCOVID dataset, attaining state-of-the-art results, not only outperforming the baseline performance but also being robust to external validation. Moreover, exploiting XAI algorithms we figure out a hierarchy among the modalities and we extract the features' intra-modality importance, enriching the trust on the predictions made by the model.

Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes

TL;DR

This work addresses prognosis of COVID-19 by fusing chest X-ray and clinical data with a novel Pareto multi-objective framework that selects and combines unimodal models across modalities. It introduces a joint–late fusion scheme that concatenates unimodal outputs and trains an end-to-end classifier, guided by both accuracy and diversity objectives. The approach achieves state-of-the-art results on the AIforCOVID dataset and shows robustness to external validation, while enabling XAI-based interpretation that reveals modality hierarchy and intra-modality feature importance. The findings demonstrate how automated model- and modality-selection can yield interpretable, high-performance multimodal predictions with practical implications for clinical risk stratification.

Abstract

The COVID-19 pandemic has caused millions of cases and deaths and the AI-related scientific community, after being involved with detecting COVID-19 signs in medical images, has been now directing the efforts towards the development of methods that can predict the progression of the disease. This task is multimodal by its very nature and, recently, baseline results achieved on the publicly available AIforCOVID dataset have shown that chest X-ray scans and clinical information are useful to identify patients at risk of severe outcomes. While deep learning has shown superior performance in several medical fields, in most of the cases it considers unimodal data only. In this respect, when, which and how to fuse the different modalities is an open challenge in multimodal deep learning. To cope with these three questions here we present a novel approach optimizing the setup of a multimodal end-to-end model. It exploits Pareto multi-objective optimization working with a performance metric and the diversity score of multiple candidate unimodal neural networks to be fused. We test our method on the AIforCOVID dataset, attaining state-of-the-art results, not only outperforming the baseline performance but also being robust to external validation. Moreover, exploiting XAI algorithms we figure out a hierarchy among the modalities and we extract the features' intra-modality importance, enriching the trust on the predictions made by the model.
Paper Structure (25 sections, 15 equations, 4 figures, 3 tables)

This paper contains 25 sections, 15 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Multimodal Deep Learning
  4. DL and COVID-19
  5. XAI
  6. Materials
  7. Methods
  8. Multimodal Learning
  9. Unimodal Learning
  10. Optimization
  11. Joint-Late Fusion
  12. Classification Layer
  13. XAI
  14. Experimental Configuration
  15. Data Pre-processing
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Multimodal joint-fusion methods: a) concatenation method bib:nojavanasghari2016deep b) multiplicative method bib:zadeh2017tensor.
  • Figure 2: Multimodal joint-late method: for each $m$ modalities, $n$ models are trained to find the optimal combination $\Gamma^*$, whose classification vector is passed to a FC neural network which will output the desired classification out of the $c$ classes.
  • Figure 3: Distribution of the performance accuracy of all the presented models and competitors in all the training configuration scenarios.
  • Figure 4: XAI modality algorithms: a) feature importance of Integrated Gradients of an instance applied to MLP-2; b) activation maps of Grad-CAM on GoogLeNet, VGG13-BN, and ResNeXt50 which are combined resulting in the weighted activation map