Multimodal Datasets with Controllable Mutual Information

TL;DR

This work tackles the challenge of understanding and benchmarking how mutual information between multiple modalities influences multimodal self-supervised learning. It proposes a three-stage framework $\mathbf{u} \to \mathbf{z} \to \mathbf{x}$ that uses a linear causal construction to produce latent variables with known MI, then applies invertible flow-based mappings to generate realistic multimodal data while preserving that MI. The key contributions are: (i) analytic or semi-analytic MI expressions for latent variables, (ii) a flow-based MI-preserving data-generation pipeline, (iii) templates to control how information is distributed across modalities for targeted ablations, and (iv) concrete dataset examples including a black-hole imaging scenario and scalable massively multimodal data, with reproducible code. The resulting datasets provide a principled, controllable testbed for MI estimators and for studying the role of information overlap in multimodal SSL as the number of modalities grows, with practical impact on benchmarking and model design.

Abstract

We introduce a framework for generating highly multimodal datasets with explicitly calculable mutual information between modalities. This enables the construction of benchmark datasets that provide a novel testbed for systematic studies of mutual information estimators and multimodal self-supervised learning techniques. Our framework constructs realistic datasets with known mutual information using a flow-based generative model and a structured causal framework for generating correlated latent variables.

Figures (4)

• Figure 1: Schematic of our dataset generation framework. a) An example DAG showing the linear mixing of proto-latents $\mathbf{u}$ via coefficients $\mathbf{\eta}, \mathbf{\rho}$ into interpretable correlated Gaussian latent variables $\mathbf{z}$. b) Overview of sampling from a multidimensional Gaussian to draw latent inputs $z_1$, $z_2$, and $z_\theta$ that are fed into invertible maps $f_1$ and $f_2$ to a realistic feature space.
• Figure 2: Eight examples of correlated pairs of images $(\mathbf{x}_1,\mathbf{x}_2)$ generated from our procedure representing two realistic modalities. In this case, both modalities are images, but corresponding to flows conditioned on different class labels ("automobile", "frog") from CIFAR-10 cifar10. The dimensionality of the data in both cases is $d=32\times 32 \times 3 = 3072$.
• Figure 3: Examples of causal structures with corresponding linear structural equations that induce specific mutual information.
• Figure 4: Information between image modalities $X_1$ and $X_i$ decreases as the distance between $X_1$ and $X_i$ increases. This, as well as the information between image modality $X_i$ and the parameter $\theta$, decreases as the total number of modalities increases.