Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HOT-POT: Optimal Transport for Sparse Stereo Matching

Antonin Clerc, Michael Quellmalz, Moritz Piening, Philipp Flotho, Gregor Kornhardt, Gabriele Steidl

TL;DR

This work tackles sparse stereo matching under occlusions and cross-modal noise by formulating camera-projected keypoints as (half)lines and solving an OT-based assignment problem. It introduces two geometry-driven costs, the 3D ray distance $d^{\mathrm{ray}}$ and the epipolar distance $d^{\mathrm{epi}}$, and enhances robustness with a depth-regularized variant of the ray distance. To handle entire objects, it extends OT to a hierarchical framework (HOT) and a HOT-POT variant for partial object-to-object matching, demonstrated on synthetic and real RGB–Thermal facial landmark data. The results show that the ray-based costs, especially with regularization, yield more accurate pointwise and objectwise correspondences and robust 3D reconstructions, highlighting practical benefits for cross-modal facial analysis and sparse stereo tasks.

Abstract

Stereo vision between images faces a range of challenges, including occlusions, motion, and camera distortions, across applications in autonomous driving, robotics, and face analysis. Due to parameter sensitivity, further complications arise for stereo matching with sparse features, such as facial landmarks. To overcome this ill-posedness and enable unsupervised sparse matching, we consider line constraints of the camera geometry from an optimal transport (OT) viewpoint. Formulating camera-projected points as (half)lines, we propose the use of the classical epipolar distance as well as a 3D ray distance to quantify matching quality. Employing these distances as a cost function of a (partial) OT problem, we arrive at efficiently solvable assignment problems. Moreover, we extend our approach to unsupervised object matching by formulating it as a hierarchical OT problem. The resulting algorithms allow for efficient feature and object matching, as demonstrated in our numerical experiments. Here, we focus on applications in facial analysis, where we aim to match distinct landmarking conventions.

HOT-POT: Optimal Transport for Sparse Stereo Matching

TL;DR

This work tackles sparse stereo matching under occlusions and cross-modal noise by formulating camera-projected keypoints as (half)lines and solving an OT-based assignment problem. It introduces two geometry-driven costs, the 3D ray distance and the epipolar distance , and enhances robustness with a depth-regularized variant of the ray distance. To handle entire objects, it extends OT to a hierarchical framework (HOT) and a HOT-POT variant for partial object-to-object matching, demonstrated on synthetic and real RGB–Thermal facial landmark data. The results show that the ray-based costs, especially with regularization, yield more accurate pointwise and objectwise correspondences and robust 3D reconstructions, highlighting practical benefits for cross-modal facial analysis and sparse stereo tasks.

Abstract

Stereo vision between images faces a range of challenges, including occlusions, motion, and camera distortions, across applications in autonomous driving, robotics, and face analysis. Due to parameter sensitivity, further complications arise for stereo matching with sparse features, such as facial landmarks. To overcome this ill-posedness and enable unsupervised sparse matching, we consider line constraints of the camera geometry from an optimal transport (OT) viewpoint. Formulating camera-projected points as (half)lines, we propose the use of the classical epipolar distance as well as a 3D ray distance to quantify matching quality. Employing these distances as a cost function of a (partial) OT problem, we arrive at efficiently solvable assignment problems. Moreover, we extend our approach to unsupervised object matching by formulating it as a hierarchical OT problem. The resulting algorithms allow for efficient feature and object matching, as demonstrated in our numerical experiments. Here, we focus on applications in facial analysis, where we aim to match distinct landmarking conventions.
Paper Structure (16 sections, 1 theorem, 40 equations, 10 figures, 6 tables, 1 algorithm)

This paper contains 16 sections, 1 theorem, 40 equations, 10 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Distances of Projected Points
  3. 3D Ray Distance
  4. Epipolar Distance
  5. Optimal Transport
  6. Point Matching via Optimal Transport
  7. Object Matching via Hierarchical Optimal Transport
  8. Algorithmic Considerations
  9. OT Algorithms
  10. Evaluation Criteria
  11. Numerical Experiments
  12. Synthetic Faces Dataset
  13. Synthetic Spheres Simulation
  14. Matching RGB and Thermal Landmarks
  15. Matching RGB and Thermal Faces
  16. ...and 1 more sections

Key Result

Proposition 2.2

If $w\in\mathcal{W}$ and $x,y\in\mathbb P^2$ are given by eq:model, then $d^\mathrm{ray}(x,y)=0$. Conversely, if $x,y\in\mathbb P^2$ and $d^\mathrm{ray}(x,y)=0$, there exists $w\in{\mathbb{R}}^3$ such that eq:model holds for some $\lambda_l,\lambda_r\in{\mathbb{R}}$. If additionally $r_x \times r_y

Figures (10)

  • Figure 1: RGB (left, middle) and thermal (right) images with facial landmarks. Landmarks conventions vary in terms of size and locations, as illustrated by the thermal image. Images were originally reported in flotho2023lagrangian.
  • Figure 2: 3D point $w$ observed by two cameras, specifically at $x$ in the left and $y$ in the right camera. The epipolar line (solid blue) in the right camera corresponding to $x$ consists of all points $y'$ that originate from 3D points $w'$ that are projected to $x$ in the left camera. In teal are the focal points $0$ and $-R^\top t$.
  • Figure 3: 3D landmarks $w$ of four faces.
  • Figure 4: 2D camera images of the four faces dataset from Figure \ref{['fig:3d_full']}.
  • Figure 5: Projection of 20 points from synthetic faces 1 (red) and 3 (blue) onto the right camera and epipolar lines of face 1, illustrating the difficulty of distinguishing faces with $d^{\mathrm{epi}}$ when two points are on a line.
  • ...and 5 more figures

Theorems & Definitions (9)

  • Remark 2.1
  • Proposition 2.2
  • proof
  • Remark 2.3: Depth-regularized Ray Distance
  • Remark 2.4: Invariance to Rotations
  • Remark 2.5
  • Remark 2.6: Epipolar rays
  • Remark 4.1: Projecting onto Binary Matrices
  • Remark 4.2: Naive Matching