Generalized Closed-form Formulae for Feature-based Subpixel Alignment in Patch-based Matching

TL;DR

This work tackles subpixel disparity refinement in patch-based matching by deriving closed-form image-space interpolation formulae for common dissimilarities and extending them from 1D to multidimensional search spaces. It presents exact solutions for NCC, SSD, and SAD in the unidimensional case and introduces generalized, barycentric- and predictive-interpolation methods for higher dimensions, with a practical algorithmic framework. Experiments on Middlebury stereo and optical-flow datasets show that feature-space interpolation, particularly with ZNCC, often yields the best subpixel accuracy and robustness to pixel-locking, especially in 2D searches. The results indicate substantial practical benefits for stereo and optical-flow pipelines, and the authors provide an open-source library LibStevi to facilitate adoption.

Abstract

Cost-based image patch matching is at the core of various techniques in computer vision, photogrammetry and remote sensing. When the subpixel disparity between the reference patch in the source and target images is required, either the cost function or the target image have to be interpolated. While cost-based interpolation is the easiest to implement, multiple works have shown that image based interpolation can increase the accuracy of the subpixel matching, but usually at the cost of expensive search procedures. This, however, is problematic, especially for very computation intensive applications such as stereo matching or optical flow computation. In this paper, we show that closed form formulae for subpixel disparity computation for the case of one dimensional matching, e.g., in the case of rectified stereo images where the search space is of one dimension, exists when using the standard NCC, SSD and SAD cost functions. We then demonstrate how to generalize the proposed formulae to the case of high dimensional search spaces, which is required for unrectified stereo matching and optical flow extraction. We also compare our results with traditional cost volume interpolation formulae as well as with state-of-the-art cost-based refinement methods, and show that the proposed formulae bring a small improvement over the state-of-the-art cost-based methods in the case of one dimensional search spaces, and a significant improvement when the search space is two dimensional.

Figures (10)

• Figure 1: In patch-based matching, two approaches can be used to achieve subpixel precision: computing the cost volume at discreet intervals and then interpolating (\ref{['fig:approaches:costInterp']}) or interpolating in the image space to directly compute a continuous cost volume (\ref{['fig:approaches:imgInterp']}). The former approach is generally preferred for its simplicity, even though the latter is more accurate and suffer from less biases.
• Figure 2: Different cost curves for different cost volume interpolation methods: image space vs cost volume space (example extracted from a real image).
• Figure 3: The pixel-locking effect, observed in cost volume-based interpolation methods (\ref{['fig:fract_part_correllation:cost']}), occurs when the expected disparity error varies with the disparity fractional part. However, it does not occur with image-based interpolation (\ref{['fig:fract_part_correllation:image']}).
• Figure 4: Visual representation of the patching operator, when used to extract a feature vector from an image patch.
• Figure 5: Generic pipeline for local patch-based matching.