Booster: a Benchmark for Depth from Images of Specular and Transparent Surfaces

TL;DR

Booster presents a high-resolution depth dataset tailored to non-Lambertian surfaces (specular and transparent) and large untextured regions, generated via a deep space-time stereo pipeline that fuses multiple textured views with RAFT-Stereo to produce dense sub-pixel ground-truth disparities and material masks. It establishes three benchmarks—Balanced Stereo, Unbalanced Stereo, and Monocular—across 606 labeled scenes (plus 15K unlabeled samples), with careful calibration, textured acquisition, super-resolution sharpening, manual cleaning, and cross-modal accuracy validation including LiDAR comparisons. Experiments reveal that state-of-the-art stereo and monocular networks struggle with Booster’s challenging materials and high resolution, though fine-tuning on Booster data yields notable improvements, especially for challenging materials. The work highlights open challenges in non-Lambertian depth estimation and motivates future directions toward outdoor data, Nerf-based ground-truth, and multi-depth-layer acquisitions to further advance depth perception in real-world, non-Lambertian scenarios.

Abstract

Estimating depth from images nowadays yields outstanding results, both in terms of in-domain accuracy and generalization. However, we identify two main challenges that remain open in this field: dealing with non-Lambertian materials and effectively processing high-resolution images. Purposely, we propose a novel dataset that includes accurate and dense ground-truth labels at high resolution, featuring scenes containing several specular and transparent surfaces. Our acquisition pipeline leverages a novel deep space-time stereo framework, enabling easy and accurate labeling with sub-pixel precision. The dataset is composed of 606 samples collected in 85 different scenes, each sample includes both a high-resolution pair (12 Mpx) as well as an unbalanced stereo pair (Left: 12 Mpx, Right: 1.1 Mpx), typical of modern mobile devices that mount sensors with different resolutions. Additionally, we provide manually annotated material segmentation masks and 15K unlabeled samples. The dataset is composed of a train set and two test sets, the latter devoted to the evaluation of stereo and monocular depth estimation networks. Our experiments highlight the open challenges and future research directions in this field.

Figures (14)

• Figure 2: Dataset acquisition overview. Our collection pipeline is made of three, main phases. Left (orange): offline calibration of our trinocular rig and the two stereo systems $L-C$ and $L-R$. Middle (green): image acquisition without ground-truth. Right (grey): acquisition of textured images, used to obtain ground-truth labels.
• Figure 3: Cameras setup and acquisition. On the left: i) passive stereo pairs collection, ii) painting of reflective/transparent materials, iii) textured stereo pairs acquisition. On the right, top-view of our camera rig, with $L$ and $R$ being two 12 Mpx sensors, and $C$ a wide-angle 2.3 Mpx sensor.
• Figure 4: Balanced/unbalanced rectification examples. From left to right: $L$, $R$, and $C$ raw images from our rig; $L_{LR}$,$R_{LR}$ rectified balanced stereo pair from the $L-R$ setup; $L_{LC}$,$C_{LC}$ unbalanced rectified stereo pair from the $L-C$ setup.
• Figure 5: Data annotation pipeline. From left to right: RGB reference image (top) and material segmentation (bottom), disparity maps (top) and point clouds (bottom) obtained by RAFT-Stereo on the passive pairs, followed by those produced by our deep space-time stereo framework, the super-resolution & sharpening procedure, and the final, manual cleaning.
• Figure 6: Cross-verification with Intel L515 LiDAR. Front (columns 1-2) and side (columns 3-4) 3D visualization of a sample scene. Columns 1,3: point cloud acquired by a LiDAR sensor. Columns 2,4: point cloud reconstructed by our framework.