Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Structure Preservation Learning

Chang Liu, Jieshi Chen, Lee H. Harrison, Artur Dubrawski

TL;DR

The results show that MSPL can imbue the learned features with external structures and help reap the beneficial synergies occurring across disparate data modalities.

Abstract

When selecting data to build machine learning models in practical applications, factors such as availability, acquisition cost, and discriminatory power are crucial considerations. Different data modalities often capture unique aspects of the underlying phenomenon, making their utilities complementary. On the other hand, some sources of data host structural information that is key to their value. Hence, the utility of one data type can sometimes be enhanced by matching the structure of another. We propose Multimodal Structure Preservation Learning (MSPL) as a novel method of learning data representations that leverages the clustering structure provided by one data modality to enhance the utility of data from another modality. We demonstrate the effectiveness of MSPL in uncovering latent structures in synthetic time series data and recovering clusters from whole genome sequencing and antimicrobial resistance data using mass spectrometry data in support of epidemiology applications. The results show that MSPL can imbue the learned features with external structures and help reap the beneficial synergies occurring across disparate data modalities.

Multimodal Structure Preservation Learning

TL;DR

The results show that MSPL can imbue the learned features with external structures and help reap the beneficial synergies occurring across disparate data modalities.

Abstract

When selecting data to build machine learning models in practical applications, factors such as availability, acquisition cost, and discriminatory power are crucial considerations. Different data modalities often capture unique aspects of the underlying phenomenon, making their utilities complementary. On the other hand, some sources of data host structural information that is key to their value. Hence, the utility of one data type can sometimes be enhanced by matching the structure of another. We propose Multimodal Structure Preservation Learning (MSPL) as a novel method of learning data representations that leverages the clustering structure provided by one data modality to enhance the utility of data from another modality. We demonstrate the effectiveness of MSPL in uncovering latent structures in synthetic time series data and recovering clusters from whole genome sequencing and antimicrobial resistance data using mass spectrometry data in support of epidemiology applications. The results show that MSPL can imbue the learned features with external structures and help reap the beneficial synergies occurring across disparate data modalities.

Paper Structure

This paper contains 25 sections, 11 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Multimodal Structure Preservation Learning.
  5. Baseline models.
  6. Cluster evaluation.
  7. Datasets
  8. Synth-TS.
  9. Proprietary dataset.
  10. DRIAMS data.
  11. Experiments and Results
  12. Uncovering latent structure in Synth-TS.
  13. Recovering WGS clusters in the proprietary dataset.
  14. Recovering AMR clusters in DRIAMS.
  15. Discussion
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Overview of the MSPL framework.
  • Figure 2: Generating the Synth-TS dataset.
  • Figure 3: Bipartite graph of Klebsiella pneumoniae clusters.
  • Figure 4: Multidimensional scaling projection based on predicted distance matrix of DRIAMS data using MSPL model, colored in ground truth clusters (left) and predicted clusters (right) respectively. MSPL recovered most of the clusters in ground truth.
  • Figure 5: Lift in F1 score for different species in the proprietary dataset and DRIAMS, sorted by the entropy of ground truth clusters or the pretext accuracy. The dot size reflects the number of MALDI samples in that species. Species label descriptions are listed in Appendix \ref{['species_label']}.
  • ...and 2 more figures