Multi-view Disparity Estimation Using a Novel Gradient Consistency Model

TL;DR

This work tackles variational disparity estimation by addressing the limited validity range of the local brightness constancy linearisation. It introduces Gradient Consistency, a data-driven weighting scheme that self‑schedules the data term across multiple views and scales by incorporating gradient and scale reliability estimates, including a physical coupling between views. The Gradient Consistency Model (GCM) derives weights from an Euler–Lagrange solution and augments them with gradient‑ and scale‑inconsistency terms, yielding a robust, self‑adjusting energy that uses an $L^1$ data term and Total Variation regularisation. Empirical results on synthetic 4D Lightfield data and real Middlebury 2006 data show that GCM outperforms coarse‑to‑fine and progressive view inclusion in both convergence rate and accuracy, with clear improvements near object boundaries and strong insensitivity to the regularisation parameter.

Abstract

Variational approaches to disparity estimation typically use a linearised brightness constancy constraint, which only applies in smooth regions and over small distances. Accordingly, current variational approaches rely on a schedule to progressively include image data. This paper proposes the use of Gradient Consistency information to assess the validity of the linearisation; this information is used to determine the weights applied to the data term as part of an analytically inspired Gradient Consistency Model. The Gradient Consistency Model penalises the data term for view pairs that have a mismatch between the spatial gradients in the source view and the spatial gradients in the target view. Instead of relying on a tuned or learned schedule, the Gradient Consistency Model is self-scheduling, since the weights evolve as the algorithm progresses. We show that the Gradient Consistency Model outperforms standard coarse-to-fine schemes and the recently proposed progressive inclusion of views approach in both rate of convergence and accuracy.

Figures (11)

• Figure 1: In this diagram, we detail how we apply the Gradient Consistency Model (GCM) to a multi-scale context. We use multiple views and multiple scales simultaneously to estimate the disparity field. Each scale and view is weighted using the GCM and then the data is passed to the optimisation stage which determines the disparity field.
• Figure 2: A diagram of the region $D(\mathbf{s})$ in the image domain $\Omega$. One can think of the individual gradients $\nabla I_{i}(\mathbf{s})$ as observations of the underlying gradient of the image surface $\nabla I(\mathbf{s})$. For the local linearisation to be valid between $I_0$ and $I_2$, the gradient must be constant along the dotted line. To assess the validity of the linearisation, we can look at the gradients $\nabla I_{0}(\mathbf{s})$, $\nabla I_{1}(\mathbf{s})$ and $\nabla I_{2}(\mathbf{s})$; if any of these gradients are different, then the linearisation is invalid.
• Figure 3: This diagram illustrates how we consider the scales in this treatment. $I_p$ is the original image data from the $p$th image. We filter the original image data using $G_{\sigma_0^\prime}$ to produce the reference scale, $q=0$ of the $p$th image, $I_{p,0}$. Different scales, $q \neq 0$ can be produced by filtering $I_p$ with $G_{\sigma_q^\prime}$ or by filtering $I_{p,0}$ with $G_{\sigma_q}$.
• Figure 4: Plots of the RMSE vs. the number of $A\mathbf{x} = \mathbf{b}$ solves (log scale) for the average of the 6-fold cross-validation on a synthetic 4D Lightfield Dataset Honauer2016. Here we compare the naive approach (purple) with the Gradient Consistency Model (orange) and the progressive inclusion of views (brown).
• Figure 5: On the top row, we have the estimated disparity fields from the three methods considered. All plots in the top row share a common colour scale. On the bottom row, we have the absolute error in the estimated disparity. All plots in the bottom row share a common scale where, black is 0 and white is $(w_\text{max} - w_\text{min})/2$. Note that, $w_\text{max}$ is the maximum and $w_\text{min}$ is the minimum of the ground truth disparity field.