On the Benefits of Visual Stabilization for Frame- and Event-based Perception

TL;DR

This paper addresses perception under large rotational motion by introducing a processing-based stabilization that leverages attitude data to decouple rotation from translation for both event- and frame-based sensors. It presents a unified stabilization framework with two pipelines (event-based IWEs/TS and frame-based stabilization via a homography) and uses an ERL-V velocity estimator to recover camera translation. The experiments show that stabilization improves feature-tracking accuracy by $27.37\%$ and ego-motion estimation accuracy by $34.82\%$, while reducing processing time by at least $25\%$. The results suggest practical benefits for robotics where payload constraints prevent mechanical stabilization, enabling faster, more reliable perception with existing IMU or attitude sensors.

Abstract

Vision-based perception systems are typically exposed to large orientation changes in different robot applications. In such conditions, their performance might be compromised due to the inherent complexity of processing data captured under challenging motion. Integration of mechanical stabilizers to compensate for the camera rotation is not always possible due to the robot payload constraints. This paper presents a processing-based stabilization approach to compensate the camera's rotational motion both on events and on frames (i.e., images). Assuming that the camera's attitude is available, we evaluate the benefits of stabilization in two perception applications: feature tracking and estimating the translation component of the camera's ego-motion. The validation is performed using synthetic data and sequences from well-known event-based vision datasets. The experiments unveil that stabilization can improve feature tracking and camera ego-motion estimation accuracy in 27.37% and 34.82%, respectively. Concurrently, stabilization can reduce the processing time of computing the camera's linear velocity by at least 25%. Code is available at https://github.com/tub-rip/visual_stabilization

Figures (6)

• Figure 1: Illustration of visual stabilization applied to images under a dominant rotational motion. The green arrows (optical flow) indicate the magnitude of the motion, which is largely compensated by our orientation stabilization approach. $\mathtt{R}(q_{C_{k}}^{\star})$ is the necessary rotation to stabilize frame $k$ (\ref{['sec:method:frames']}).
• Figure 2: Block diagrams of the proposed stabilization approach, applied to estimation of the camera's linear velocity.
• Figure 3: Data in the stabilization processing pipeline of \ref{['fig:block_diagram:events']} (\ref{['subsec:event_warpping']}), illustrated using a sequence from Zhu18ral.
• Figure 4: Visualization of a frame (left) and an IWE (right) after stabilization. 38 Data from VECtor3 sequences Gao22ral.
• Figure 5: Velocity plots using ERL-V with IWEs on sequence ESIM 3. Overall, estimation is good, producing small errors. The red curves correspond to the ground truth velocities, while the blue curves describe the estimated $\mathbf{V}$. The green dotted curves represent the camera angular velocities over time.