Multi-modal data generation with a deep metric variational autoencoder

TL;DR

The paper tackles conditional multi-modal data generation when modalities lack direct pixel correspondence, using a deep metric VAE with triplet loss to create clustered latent representations. The proposed architecture comprises two encoders and two decoders that converge on a common $128$-dimensional latent space, enabling generation of paired otoscopy images and WBT measurements conditioned by class via latent-space sampling from KDE-estimated distributions. Training integrates SSIM for image reconstruction, BCE for WBT, and a triplet loss with KL regularization, producing realistic, class-consistent multi-modal pairs and supporting data augmentation while preserving privacy. The results demonstrate discernible class clusters in latent space, plausible conditional generations across three otitis media classes, and potential applicability to other biomedical multi-modal datasets, albeit with some image blur that invites future fidelity improvements.

Abstract

We present a deep metric variational autoencoder for multi-modal data generation. The variational autoencoder employs triplet loss in the latent space, which allows for conditional data generation by sampling in the latent space within each class cluster. The approach is evaluated on a multi-modal dataset consisting of otoscopy images of the tympanic membrane with corresponding wideband tympanometry measurements. The modalities in this dataset are correlated, as they represent different aspects of the state of the middle ear, but they do not present a direct pixel-to-pixel correlation. The approach shows promising results for the conditional generation of pairs of images and tympanograms, and will allow for efficient data augmentation of data from multi-modal sources.

Figures (5)

• Figure 1: Examples from the dataset and generated examples: otoscopy images (top) and WBT measurements (bottom). Acute otitis media (left two images), otitis media with effusion (middle two images), no effusion (right two images).
• Figure 2: Structure of the multi-modal triplet VAE. Top figure shows the overall structure with two encoders, concatenation of the outputs, sampling, and two decoders. Bottom figure shows the residual blocks used in both encoders and decoders. BN refers to batch normalization.
• Figure 3: t-SNE visualization of test data latent embeddings.
• Figure 4: Examples of generated otoscopy images. Top row: AOM, middle row: OME, bottom row: NOE. Best viewed with zoom.
• Figure 5: Overview of generated WBT measurements. Top row: AOM, middle row: OME, bottom row: NOE. Best viewed with zoom.