Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

STORM: Strategic Orchestration of Modalities for Rare Event Classification

Payal Kamboj, Ayan Banerjee, Sandeep K. S. Gupta

TL;DR

This paper tackles the challenge of multimodal learning for rare-event classification in biomedicine by proposing STORM, an entropy-based modality selection framework that combines expert-guided narrowing with data-driven assessment. It quantifies information content via class-wise entropy and employs an entropy-imbalance metric and information gain to rank and cascade modalities, addressing NP-hard subset selection through a Hunt-style procedure. The method is demonstrated on seizure onset zone detection from resting-state fMRI, revealing that expert-derived spatial features (D1) can be crucial while temporal features (D2) may contribute less to AI performance, and that the basic modality is essential. The work advances data-efficient, clinically applicable AI by enabling principled modality orchestration and could generalize to other rare-event biomedical tasks requiring careful modality budgeting.

Abstract

In domains such as biomedical, expert insights are crucial for selecting the most informative modalities for artificial intelligence (AI) methodologies. However, using all available modalities poses challenges, particularly in determining the impact of each modality on performance and optimizing their combinations for accurate classification. Traditional approaches resort to manual trial and error methods, lacking systematic frameworks for discerning the most relevant modalities. Moreover, although multi-modal learning enables the integration of information from diverse sources, utilizing all available modalities is often impractical and unnecessary. To address this, we introduce an entropy-based algorithm STORM to solve the modality selection problem for rare event. This algorithm systematically evaluates the information content of individual modalities and their combinations, identifying the most discriminative features essential for rare class classification tasks. Through seizure onset zone detection case study, we demonstrate the efficacy of our algorithm in enhancing classification performance. By selecting useful subset of modalities, our approach paves the way for more efficient AI-driven biomedical analyses, thereby advancing disease diagnosis in clinical settings.

STORM: Strategic Orchestration of Modalities for Rare Event Classification

TL;DR

This paper tackles the challenge of multimodal learning for rare-event classification in biomedicine by proposing STORM, an entropy-based modality selection framework that combines expert-guided narrowing with data-driven assessment. It quantifies information content via class-wise entropy and employs an entropy-imbalance metric and information gain to rank and cascade modalities, addressing NP-hard subset selection through a Hunt-style procedure. The method is demonstrated on seizure onset zone detection from resting-state fMRI, revealing that expert-derived spatial features (D1) can be crucial while temporal features (D2) may contribute less to AI performance, and that the basic modality is essential. The work advances data-efficient, clinically applicable AI by enabling principled modality orchestration and could generalize to other rare-event biomedical tasks requiring careful modality budgeting.

Abstract

In domains such as biomedical, expert insights are crucial for selecting the most informative modalities for artificial intelligence (AI) methodologies. However, using all available modalities poses challenges, particularly in determining the impact of each modality on performance and optimizing their combinations for accurate classification. Traditional approaches resort to manual trial and error methods, lacking systematic frameworks for discerning the most relevant modalities. Moreover, although multi-modal learning enables the integration of information from diverse sources, utilizing all available modalities is often impractical and unnecessary. To address this, we introduce an entropy-based algorithm STORM to solve the modality selection problem for rare event. This algorithm systematically evaluates the information content of individual modalities and their combinations, identifying the most discriminative features essential for rare class classification tasks. Through seizure onset zone detection case study, we demonstrate the efficacy of our algorithm in enhancing classification performance. By selecting useful subset of modalities, our approach paves the way for more efficient AI-driven biomedical analyses, thereby advancing disease diagnosis in clinical settings.

Paper Structure

This paper contains 12 sections, 6 equations, 2 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Expert Knowledge based Modality Selection
  3. Overview of Solution
  4. Application Domain
  5. Methodology
  6. Modality Assessment
  7. Modality Selection
  8. Data and Results
  9. Data collection
  10. Experiments and Results
  11. Another Case Study
  12. Conclusions

Figures (2)

  • Figure 1: Modality Selection: Within this framework, specific modalities, such as Modality 2 and Modality 4 as illustrated, are initially curated by domain experts. To assess the discriminative potential of each modality, an entropy-based technique is employed for selection. Furthermore, the remaining modalities can also undergo this evaluation to determine their respective contributions.
  • Figure 2: SOZ classification using STORM.