Mutual Information Analysis in Multimodal Learning Systems

TL;DR

Problem: quantify how much information is shared between modalities in multimodal learning and how that sharing affects downstream tasks. Approach: introduce InfoMeter, an entropy-based MI estimator built on invertible transforms and entropy models, and apply it to a camera-LiDAR fusion system for 3D object detection. Key findings: across four schemes on nuScenes with FUTR3D, lower MI between camera and LiDAR ($I(X;Y) = H(X) + H(Y) - H(X,Y)$) correlates with higher mAP, supporting a redundancy interpretation over reinforcement. Significance: these results motivate data diversification across modalities during training and provide a practical tool for diagnosing and improving multimodal systems.

Abstract

In recent years, there has been a significant increase in applications of multimodal signal processing and analysis, largely driven by the increased availability of multimodal datasets and the rapid progress in multimodal learning systems. Well-known examples include autonomous vehicles, audiovisual generative systems, vision-language systems, and so on. Such systems integrate multiple signal modalities: text, speech, images, video, LiDAR, etc., to perform various tasks. A key issue for understanding such systems is the relationship between various modalities and how it impacts task performance. In this paper, we employ the concept of mutual information (MI) to gain insight into this issue. Taking advantage of the recent progress in entropy modeling and estimation, we develop a system called InfoMeter to estimate MI between modalities in a multimodal learning system. We then apply InfoMeter to analyze a multimodal 3D object detection system over a large-scale dataset for autonomous driving. Our experiments on this system suggest that a lower MI between modalities is beneficial for detection accuracy. This new insight may facilitate improvements in the development of future multimodal learning systems.

Figures (7)

• Figure 1: Block diagram of the InfoMeter for estimating the MI between sources $X$ and $Y$. $C$ represents concatenation, $\phi$'s are invertible transformations and $h$'s are entropy estimators.
• Figure 2: The simplified architecture of FUTR3D. In our experiments, $X$ and $Y$ are camera and LiDAR features after feature sampling, which are two single-channel feature maps of the same spatial size. When using 600 queries with an embedding length of 256, these two maps are of size $1\times 600\times 256$.
• Figure 3: Two visual samples from the datasets generated in Scheme 1 showing a low-clutter scene from the first dataset (top) and a high-clutter scene from the second dataset (bottom) along with their ground-truth bounding boxes. Note that the camera modality in nuScenes consists of 6 cameras.
• Figure 4: A sample image from the dataset generated in Scheme 2. Left: the original image; Middle: the masked image; Right: the inpainted result. As seen from this example, the generative inpainting method used in Scheme 2 was able to effectively reconstruct the masked car.
• Figure 5: An example demonstrating the possible emergence of spurious objects when using a generative inpainting method. Left: the original image; Right: the inpainted result using Scheme 2. The generated spurious objects are highlighted by the red boxes.