Motion and Structure from Event-based Normal Flow

TL;DR

It is shown that the event-based normal flow can be used, via the proposed geometric error term, as an alternative to the full flow in solving a family of geometric problems that involve instantaneous first-order kinematics and scene geometry.

Abstract

Recovering the camera motion and scene geometry from visual data is a fundamental problem in the field of computer vision. Its success in standard vision is attributed to the maturity of feature extraction, data association and multi-view geometry. The recent emergence of neuromorphic event-based cameras places great demands on approaches that use raw event data as input to solve this fundamental problem. Existing state-of-the-art solutions typically infer implicitly data association by iteratively reversing the event data generation process. However, the nonlinear nature of these methods limits their applicability in real-time tasks, and the constant-motion assumption leads to unstable results under agile motion. To this end, we rethink the problem formulation in a way that aligns better with the differential working principle of event cameras. We show that the event-based normal flow can be used, via the proposed geometric error term, as an alternative to the full flow in solving a family of geometric problems that involve instantaneous first-order kinematics and scene geometry. Furthermore, we develop a fast linear solver and a continuous-time nonlinear solver on top of the proposed geometric error term. Experiments on both synthetic and real data show the superiority of our linear solver in terms of accuracy and efficiency, and indicate its complementary feature as an initialization method for existing nonlinear solvers. Besides, our continuous-time non-linear solver exhibits exceptional capability in accommodating sudden variations in motion since it does not rely on the constant-motion assumption.

Figures (10)

• Figure 1: Overview of our event-based motion estimation approach. Normal flow vectors are computed from the raw event data and are used as input to the two solvers. The result of the linear solver consists of fitted motion models $\{\boldsymbol{\theta}_1, \boldsymbol{\theta}_2, \cdots \}$ at discrete time instants, which can be used to initialize the continuous-time nonlinear solver.
• Figure 2: Toy example of applying the proposed normal-flow constraint. (a): A 2D registration task using as input either optical flow or normal flow. The groundtruth displacement is defined by a global flow of $(1.732, -1)^\top$. (b)-(d): The loss landscape obtained using different geometric measurements in the registration task. The red dot denotes the resulting displacement, and the green cross the groundtruth displacement. The bottom-left regions in (b)-(d) display the corresponding registration results.
• Figure 3: Qualitative result of optical flow and depth on sequence three_wall_translation of our synthetic data. Note that our method returns sparse results.
• Figure 4: Angular velocity estimation using the continuous-time nonlinear solver (Ours).
• Figure 5: Illustration of event-based normal flow, explained on the spatio-temporal profile induced by a moving edge. Given a pixel coordinate $\mathbf{x}$, when the increment $\mathrm{\Delta}\mathbf{x}$'s direction $\mathbf{d}$ equals $\mathbf{d}_{\max}$ ($\mathbf{d}_{\max}$ is the one that maximizes the directional derivative $\nabla_{\mathbf{d}}\mathrm{\Sigma}_{e}(\mathbf{x})=\mathbf{d}\cdot\nabla\mathrm{\Sigma}_{e}(\mathbf{x})$, i.e., $\mathbf{d}_{\max}=\frac{\nabla\mathrm{\Sigma}_{e}(\mathbf{x})}{\|\nabla\mathrm{\Sigma}_{e}(\mathbf{x})\|}$), it aligns with the direction of the time surface gradient $\nabla\mathrm{\Sigma}_{e}(\mathbf{x})$, thereby determining the direction of the normal flow $\mathbf{n}(\mathbf{x})$. Since the time increment (lifetime) corresponding to $\mathbf{d}_{\max}$ is $\mathrm{\Delta}{t}_{\max}=\mathbf{d}_{\max}\cdot\nabla\mathrm{\Sigma}_{e}(\mathbf{x})=\|\nabla\mathrm{\Sigma}_{e}(\mathbf{x})\|$, the normal flow $\mathbf{n}(\mathbf{x})=\frac{\mathbf{d}_{\max}}{\mathrm{\Delta}{t}_{\max}}=\frac{\nabla\mathrm{\Sigma}_{e}(\mathbf{x})}{\|\nabla\mathrm{\Sigma}_{e}(\mathbf{x})\|^2}$.