Multi-Label Stereo Matching for Transparent Scene Depth Estimation

TL;DR

The paper tackles depth estimation in transparent scenes by reframing stereo matching as a multi-label regression problem. It introduces a pixel-wise multivariate Gaussian representation parameterized by a mean vector $\mu$ and covariance $\Sigma$, learned iteratively with a GRU-based framework to separate foreground and background depths. The method enables simultaneous reconstruction of both the transparent foreground and the occluded background, validated on a new TranScene dataset that includes multi-label ground truth and semantic masks. Empirical results demonstrate improved depth accuracy for transparent surfaces while preserving background geometry, offering a robust, hardware-light approach for AR/robotic perception in transparent environments.

Abstract

In this paper, we present a multi-label stereo matching method to simultaneously estimate the depth of the transparent objects and the occluded background in transparent scenes.Unlike previous methods that assume a unimodal distribution along the disparity dimension and formulate the matching as a single-label regression problem, we propose a multi-label regression formulation to estimate multiple depth values at the same pixel in transparent scenes. To resolve the multi-label regression problem, we introduce a pixel-wise multivariate Gaussian representation, where the mean vector encodes multiple depth values at the same pixel, and the covariance matrix determines whether a multi-label representation is necessary for a given pixel. The representation is iteratively predicted within a GRU framework. In each iteration, we first predict the update step for the mean parameters and then use both the update step and the updated mean parameters to estimate the covariance matrix. We also synthesize a dataset containing 10 scenes and 89 objects to validate the performance of transparent scene depth estimation. The experiments show that our method greatly improves the performance on transparent surfaces while preserving the background information for scene reconstruction. Code is available at https://github.com/BFZD233/TranScene.

Figures (6)

• Figure 1: The visualization of the multi-label disparity map and the reconstructed point cloud. The first column is the foreground disparity map and the second column is the background disparity map. The rest two columns represent the points cloud reconstructed by the multi-label disparity map and the corresponding robot navigation. Our method can estimate the depth of the transparent foreground and the occluded background simultaneously, as shown in the red and green box.
• Figure 2: The pipeline of our multi-label stereo matching method. Our multivariate Gaussian representation (MGR) is parameterized by a mean vector $\mu=(\mu_0,\mu_1)^T$ and a covariance matrix $\left[ \sigma_0\rho\sigma_0\sigma_1\rho\sigma_0\sigma_1\sigma_1 \right]$, which are iteratively predicted in a GRU framework. ⓦ represents the warping operation between left and right images.
• Figure 3: The network architecture of our multivariate Gaussian representation (MGR) prediction module. we use a two-stage strategy to sequentially predict the mean vector $\mu=(\mu_0,\mu_1)^T$ and the covariance matrix $\left[ \sigma_0^2\rho\sigma_0\sigma_1\rho\sigma_0\sigma_1\sigma_1^2 \right]$. Ⓛ is the lookup operation used to sample the cost value according to the predicted disparity. Ⓒ is the concatenation operation.
• Figure 4: The visualization of the parameters ($\mu_0,\mu_1,\rho$) of the multivariate Gaussian representation at each iteration.
• Figure 5: The visualization of the disparity map on TranScene dataset.