Eliminating Warping Shakes for Unsupervised Online Video Stitching

TL;DR

This work tackles warping shakes that arise when applying image stitching to videos captured by hand-held cameras. It introduces StabStitch, an unsupervised online framework that jointly stitches and stabilizes by deriving stitching trajectories from integrated spatial and temporal warps and then smoothing them with a dedicated warp-smoothing network trained online. Key contributions include the first unsupervised online video stitching method, a holistic StabStitch-D dataset for evaluation, and demonstrated improvements in scene robustness and inference speed over state-of-the-art approaches. The proposed system enables real-time, robust panoramic video stitching suitable for practical deployment in hand-held scenarios.

Abstract

In this paper, we retarget video stitching to an emerging issue, named warping shake, when extending image stitching to video stitching. It unveils the temporal instability of warped content in non-overlapping regions, despite image stitching having endeavored to preserve the natural structures. Therefore, in most cases, even if the input videos to be stitched are stable, the stitched video will inevitably cause undesired warping shakes and affect the visual experience. To eliminate the shakes, we propose StabStitch to simultaneously realize video stitching and video stabilization in a unified unsupervised learning framework. Starting from the camera paths in video stabilization, we first derive the expression of stitching trajectories in video stitching by elaborately integrating spatial and temporal warps. Then a warp smoothing model is presented to optimize them with a comprehensive consideration regarding content alignment, trajectory smoothness, spatial consistency, and online collaboration. To establish an evaluation benchmark and train the learning framework, we build a video stitching dataset with a rich diversity in camera motions and scenes. Compared with existing stitching solutions, StabStitch exhibits significant superiority in scene robustness and inference speed in addition to stitching and stabilization performance, contributing to a robust and real-time online video stitching system. The code and dataset are available at https://github.com/nie-lang/StabStitch.

Figures (9)

• Figure 1: The occurrence and elimination of warping shakes. Left: stable camera trajectories for input videos. Middle: warping shakes are produced by image stitching, yielding unsmooth stitching trajectories. Right: StabStitch eliminates these shakes successfully.
• Figure 2: The overview of StabStitch. We first obtain stitching trajectories by integrating spatial and temporal warps. Then the stitching trajectories are optimized by the warp smoothing model to produce unsmooth-to-smooth stitching warps.
• Figure 3: The online stitching mode. We define a sliding window to process a short sequence and display the last frame on the online screen.
• Figure 4: The proposed StabStitch-D dataset with a large diversity in camera motions and scenes. We exhibit several video pairs for each category.
• Figure 5: Qualitative comparison with Nie et al.'s video stitching nie2017dynamic on a regular case (top) and a fast-moving case (bottom). The numbers below the images indicate the time at which the frame appears in the video. Please zoom in for the best view.