xMOD: Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D motion

TL;DR

This paper tackles unsupervised multi-object discovery in 3D point clouds by leveraging 2D motion cues and bridging RGB and LiDAR data. It introduces DIOD-3D, a 3D object discovery baseline that uses front-view LiDAR projections and a 2D motion-guided scene completion pretext, then extends to xMOD, a cross-modal distillation framework where 2D and 3D teacher–student pairs exchange pseudo-labels under motion supervision. The approach demonstrates that cross-modal learning improves both modalities, with late fusion yielding the best multi-modal performance on synthetic TRI-PD and real-world KITTI and Waymo datasets (F1@50 gains of several points over 2D-only baselines); scene completion and careful fusion strategies are key to robustness. The work provides new benchmarks for 3D-available data and shows practical impact for robust, sensor-flexible object discovery in driving scenarios, highlighting the value of transferring 2D motion-informed supervision into 3D representations. Overall, xMOD advances unsupervised multi-modal object discovery by combining motion cues, scene completion, and cross-modal distillation to address modality-specific limitations and improve generalization.

Abstract

Object discovery, which refers to the task of localizing objects without human annotations, has gained significant attention in 2D image analysis. However, despite this growing interest, it remains under-explored in 3D data, where approaches rely exclusively on 3D motion, despite its several challenges. In this paper, we present a novel framework that leverages advances in 2D object discovery which are based on 2D motion to exploit the advantages of such motion cues being more flexible and generalizable and to bridge the gap between 2D and 3D modalities. Our primary contributions are twofold: (i) we introduce DIOD-3D, the first baseline for multi-object discovery in 3D data using 2D motion, incorporating scene completion as an auxiliary task to enable dense object localization from sparse input data; (ii) we develop xMOD, a cross-modal training framework that integrates 2D and 3D data while always using 2D motion cues. xMOD employs a teacher-student training paradigm across the two modalities to mitigate confirmation bias by leveraging the domain gap. During inference, the model supports both RGB-only and point cloud-only inputs. Additionally, we propose a late-fusion technique tailored to our pipeline that further enhances performance when both modalities are available at inference. We evaluate our approach extensively on synthetic (TRIP-PD) and challenging real-world datasets (KITTI and Waymo). Notably, our approach yields a substantial performance improvement compared with the 2D object discovery state-of-the-art on all datasets with gains ranging from +8.7 to +15.1 in F1@50 score. The code is available at https://github.com/CEA-LIST/xMOD

Figures (3)

• Figure 1: Overview of the proposed approach. i) DIOD-3D. At each iteration, a sequence front-view projections of point clouds is passed to the 3D teacher and student models. Attention maps from the teacher model are presented as targets to the student model through $L^{dist}_{3D\rightarrow3D}$. An MSE objective is employed to predict the original scene from input with missing data, enabling 3D scene completion as an auxiliary task for 3DOD. ii) Cross-modal distillation (xMOD). Alongside the 3D branch, sequences of RGB images are forwarded to the 2D teacher and student models. $L^{dist}_{2D\rightarrow3D}$ means pseudo-labels from the 2D teacher model are aligned with the 3D student input and used for its supervision; $L^{dist}_{3D\rightarrow2D}$ works similarly for 3D to 2D pseudo-labeling. Motion pseudo-labels $\mathcal{M}_{2D}$ and $\mathcal{M}_{3D}$ are used for regularization, with $\mathcal{M}_{3D}$ being the 2D motion segments with corresponding 3D points. We omit representing 2D reconstruction and 3D completion task for simplification.
• Figure 2: Qualitative comparison of our method with state-of-the-art approach DIOD DIOD, the cross-modal branches xMOD (2D), xMOD (3D) separately and the final result after fusion xMOD (2D+3D) in real-world scenes (KITTI KITTI). Parentheses indicate the modality used during inference. Each colored mask represents the content of one slot. The segmentations are displayed above the RGB image for visualisation purposes only. Improvements in xMOD are especially evident in pedestrian detection and background noise suppression.
• Figure 3: 3D visualization of predictions produced by xMOD (2D+3D). The background is displayed in gray and each colored mask represents the content of a distinct slot.