RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos

TL;DR

WildRGB-D tackles the scarcity of real-world depth-enabled 3D data by introducing a large-scale RGB-D object video dataset captured in the wild with 360-degree trajectories. It provides real-world scale camera poses, object masks, and aggregated point clouds across 46 categories, enabling benchmarks for novel view synthesis, camera pose estimation, surface reconstruction, and 6D pose estimation. The experiments demonstrate that depth information and RGB-D supervision consistently improve NVS quality, geometry learning, and pose estimation in diverse scenes, highlighting the dataset’s potential to advance 3D object learning in realistic environments. Limitations include the absence of object-level 6D pose annotations, which the authors plan to address, and the authors intend to release both the dataset and evaluation code for public use.

Abstract

We introduce a new RGB-D object dataset captured in the wild called WildRGB-D. Unlike most existing real-world object-centric datasets which only come with RGB capturing, the direct capture of the depth channel allows better 3D annotations and broader downstream applications. WildRGB-D comprises large-scale category-level RGB-D object videos, which are taken using an iPhone to go around the objects in 360 degrees. It contains around 8500 recorded objects and nearly 20000 RGB-D videos across 46 common object categories. These videos are taken with diverse cluttered backgrounds with three setups to cover as many real-world scenarios as possible: (i) a single object in one video; (ii) multiple objects in one video; and (iii) an object with a static hand in one video. The dataset is annotated with object masks, real-world scale camera poses, and reconstructed aggregated point clouds from RGBD videos. We benchmark four tasks with WildRGB-D including novel view synthesis, camera pose estimation, object 6d pose estimation, and object surface reconstruction. Our experiments show that the large-scale capture of RGB-D objects provides a large potential to advance 3D object learning. Our project page is https://wildrgbd.github.io/.

Figures (9)

• Figure 1: WildRGB-D Dataset contains almost 8500 recorded objects and nearly 20000 RGBD videos in 46 common categories with corresponding object masks and 3D point clouds.
• Figure 2: The camera poses trajectory in WildRGB-D Dataset. We visualize the corresponding camera in each scene of our dataset, showing that our dataset is featured in 360 degree full and dense multi-view camera poses.
• Figure 3: Statistics of WildRGB-D Dataset list the total and per-category number of objects and different types of videos.
• Figure 4: Distribution visualization of different kinds of Object 6D pose dataset and WildRGB-D dataset. We observe obvious disparity between Wild6D and Our dataset. Visualizations of wang2019normalizedwang2022phocalliu2022stereobj1mfu2022categoryleveljung2023housecat6d are from jung2023housecat6d.
• Figure 5: Point cloud reconstruction of objects in WildRGB-D Dataset. We reconstruct the aggregated point cloud of the scene by leveraging existed 3D annotations of camera poses and depth images.