Real-time Structure Flow

TL;DR

The paper introduces structure flow, a robo-centric 3D scene velocity field scaled by inverse depth, and derives PDE-based evolution equations for brightness, structure flow, and depth on a spherical camera. It then presents a GPU-accelerated predictor-update filter with a pyramidal Spherepix architecture that propagates forward in time via PDEs and refines with image and depth measurements, achieving real-time performance up to ~$600$ Hz on $512\times 512$ and ~172 Hz on $1024\times 1024$ data. Ground-truth validation on synthetic high-speed sequences and real stereo driving data demonstrates dense, high-rate motion estimation with RMSE around $0.3$ pixels and AAE around $20^{\circ}$ after convergence, confirming practical viability for robust local planning. The work advances real-time, dense, robo-centric perception by integrating a physics-inspired evolution model with an efficient, multi-scale, GPU implementation, offering significant potential for high-speed autonomous navigation and obstacle avoidance."

Abstract

This article introduces the structure flow field; a flow field that can provide high-speed robo-centric motion information for motion control of highly dynamic robotic devices and autonomous vehicles. Structure flow is the angular 3D velocity of the scene at a given pixel. We show that structure flow posses an elegant evolution model in the form of a Partial Differential Equation (PDE) that enables us to create dense flow predictions forward in time. We exploit this structure to design a predictor-update algorithm to compute structure flow in real time using image and depth measurements. The prediction stage takes the previous estimate of the structure flow and propagates it forward in time using a numerical implementation of the structure flow PDE. The predicted flow is then updated using new image and depth data. The algorithm runs up to 600 Hz on a Desktop GPU machine for 512x512 images with flow values up to 8 pixels. We provide ground truth validation on high-speed synthetic image sequences as well as results on real-life video on driving scenarios.

Figures (9)

• Figure 1: Scene $\mathbf{v}(\boldsymbol{\eta})$, structure $\mathbf{w}(\boldsymbol{\eta})$ and optical flow $\boldsymbol{\Phi}(\boldsymbol{\eta})$.
• Figure 2: Scene, structure and optical flow fields for a forward moving camera in a static scene.
• Figure 3: Pyramidal filter architecture.
• Figure 4: Spherepix grid. Spherical coordinate $\boldsymbol{\eta}_{ij}$ and its associated tangent plane $T_{\eta_{ij}}S^2$. Vector $\boldsymbol{\mu} \in T_{\eta_{ij}}S^2$ can be represented as a 2-vector $\boldsymbol{\beta}$ in the (row, column) direction.
• Figure 5: Contribution of the source terms in Equation \ref{['eq:assumption']}. The simulated camera runs at 300 Hz and moves with a linear velocity of $5 m/s$, acceleration of $1 m/s^2$ both in the $z$ direction and angular velocity of $180 \deg/s$ in $y$ axis.