Unified Vertex Motion Estimation for Integrated Video Stabilization and Stitching in Tractor-Trailer Wheeled Robots

TL;DR

This work tackles surround-view perception for tractor-trailer wheeled robots by addressing three simultaneous challenges: asynchronous vibrations, dynamic tractor-trailer poses, and large parallax. It introduces a unified Vertex Motion framework that jointly handles stabilization and stitching via mesh-vertex motion fields, combining Dual Independence Stabilization with Random Plane Stitching and enforcing a single optimization that minimizes both temporal and spatial misalignments. Key contributions include intra- and inter-unit motion decompositions, a unified energy formulation with a three-stage optimization, and real-world TTWR validation showing robust, real-time capable performance. The approach yields higher stitching fidelity and improved stability in challenging TTWR scenarios, enabling more reliable panoramic perception for large robotic platforms.

Abstract

Tractor-trailer wheeled robots need to perform comprehensive perception tasks to enhance their operations in areas such as logistics parks and long-haul transportation. The perception of these robots faces three major challenges: the asynchronous vibrations between the tractor and trailer, the relative pose change between the tractor and trailer, and the significant camera parallax caused by the large size. In this paper, we employ the Dual Independence Stabilization Motion Field Estimation method to address asynchronous vibrations between the tractor and trailer, effectively eliminating conflicting motion estimations for the same object in overlapping regions. We utilize the Random Plane-based Stitching Motion Field Estimation method to tackle the continuous relative pose changes caused by the articulated hitch between the tractor and trailer, thus eliminating dynamic misalignment in overlapping regions. Furthermore, we apply the Unified Vertex Motion Estimation method to manage the challenges posed by the tractor-trailer's large physical size, which results in severely low overlapping regions between the tractor and trailer views, thus preventing distortions in overlapping regions from exponentially propagating into non-overlapping areas. Furthermore, this framework has been successfully implemented in real tractor-trailer wheeled robots. The proposed Unified Vertex Motion Video Stabilization and Stitching method has been thoroughly tested in various challenging scenarios, demonstrating its accuracy and practicality in real-world.

Figures (12)

• Figure 1: Surround-view system of tractor-trailer wheeled robots.
• Figure 2: Overview of our method: Leveraging real-scene data from Beijing Institute of Technology, our method unfolds across five stages: (1) Pre-calibrated Semi-Surround View: Merging images from tractor and trailer cameras using pre-calibrated parameters to form forward and rearward wide-angle views; (2) Motion Initialization: Computing feature point displacements, distinguishing inter-motion and intra-motion for adjacent and same cameras at varying moments; (3) Motion Propagation: Filtering and amalgamating motions on vertices to craft a unified vertex motion; (4) Unified Stabilization and Stitching Optimization: Smoothing motion trajectories, tuning weights to balance stitching and stabilization costs; (5) Panoramic Video Generation: Integrating optimized images to craft a panoramic view.
• Figure 3: Overview of Specific Technical Aspects: In the diagram, orange symbols represent data, $\textcolor{orange}{m_{i,j}^{u,intra}}$ (intra-unit feature motion), $\textcolor{orange}{m_{i,j}^{u \to v}}$ (inter-unit correspondence motion). Black annotations describe core modules: Stabilization Motion Fields for temporal vibration compensation, Stitching Motion Fields for spatial alignment.
• Figure 4: Illustration of Intra Motion and Inter Motion: Demonstrating the distinction between Inter motion and Intra motion, with Inter motion showcasing the movement between adjacent views at a singular moment and Intra motion depicting the movement within the same view across adjacent moments.
• Figure 5: Illustration of vertex motion: This illustration delineates the concept of vertex motion, detailing the extrapolation of motion from feature points to all vertices within the ellipse.