Any2Any: Incomplete Multimodal Retrieval with Conformal Prediction

TL;DR

Any2Any is proposed-a novel retrieval framework that addresses scenarios where both query and reference instances have incomplete modalities, and achieves a Recall@5 of 35% on the KITTI dataset, which is on par with baseline models with complete modalities.

Abstract

Autonomous agents perceive and interpret their surroundings by integrating multimodal inputs, such as vision, audio, and LiDAR. These perceptual modalities support retrieval tasks, such as place recognition in robotics. However, current multimodal retrieval systems encounter difficulties when parts of the data are missing due to sensor failures or inaccessibility, such as silent videos or LiDAR scans lacking RGB information. We propose Any2Any-a novel retrieval framework that addresses scenarios where both query and reference instances have incomplete modalities. Unlike previous methods limited to the imputation of two modalities, Any2Any handles any number of modalities without training generative models. It calculates pairwise similarities with cross-modal encoders and employs a two-stage calibration process with conformal prediction to align the similarities. Any2Any enables effective retrieval across multimodal datasets, e.g., text-LiDAR and text-time series. It achieves a Recall@5 of 35% on the KITTI dataset, which is on par with baseline models with complete modalities.

Figures (6)

• Figure 1: The Any2Any retrieval framework retrieves multimodal data with varying incomplete modalities across instances. We employ a two-stage calibration process using conformal prediction to facilitate comparisons between query and reference instances, each has different incomplete modalities. Any2Any supports any number and combination of modalities, enabling it to adapt to any multimodal retrieval dataset. This illustrative figure uses data from the KITTI dataset KITTI and captions generated by LLaVA liu2023improvedllava.
• Figure 2: Normalizing similarity score distributions with first stage conformal prediction. Comparison of similarity score distributions before (blue) and after (red) calibration with conformal prediction for (a) LiDAR-to-LiDAR and (b) text-to-text retrieval tasks in KITTI. Before calibration, the similarity scores of the two modalities fall in various ranges, and after calibration, they both range between $[0, 1]$, enabling direct comparison.
• Figure 3: Conformal retrieval separates correct and incorrect distributions. (a) shows the calibration score distributions for correct and incorrect retrievals, highlighting distinct patterns. (b) shows the probability of correct retrieval increases with calibration scores. We use the KITTI dataset's calibration set here.
• Figure 4: Cross-modal retrieval performance with complete modalities. The heatmaps show retrieval performance across various query and reference modality pairs on (a) KITTI, (b) MSR-VTT, and (c) Monash Bitcoin datasets. Each cell represents the Recall@$5$ score, highlighting the variation in retrieval performance between modalities, with some modalities performing significantly better than others.
• Figure 5: Correlation coefficients between unimodal similarity scores. There are no correlations between the unimodal scores from the two encoders, regardless of whether linear or Spearman correlations are considered. Lip-loc encodes LiDAR and image data, and GTR encoders text here.