StabStitch++: Unsupervised Online Video Stitching with Spatiotemporal Bidirectional Warps

TL;DR

This paper tackles warping shake in video stitching by presenting StabStitch++, an unsupervised online framework that jointly optimizes spatial stitching and temporal stabilization through spatiotemporal bidirectional warps. It introduces a differentiable bidirectional spatial warp that projects onto a virtual middle plane, enabling balanced distribution of alignment and distortion across views, and computes stitching trajectories by integrating spatial and temporal warps. A warp-smoothing module with a hybrid loss enforces content alignment, trajectory smoothness, and online collaboration, enabling real-time online stitching without camera poses. The authors also provide the StabStitch-D dataset for training and benchmarking across diverse motions and scenes, and demonstrate that StabStitch++ achieves superior stitching performance, robustness, and efficiency compared with state-of-the-art methods. The work advances practical online video stitching by delivering a unified, unsupervised approach with real-time capabilities and robust performance on challenging scenes.

Abstract

We retarget video stitching to an emerging issue, named warping shake, which unveils the temporal content shakes induced by sequentially unsmooth warps when extending image stitching to video stitching. Even if the input videos are stable, the stitched video can inevitably cause undesired warping shakes and affect the visual experience. To address this issue, we propose StabStitch++, a novel video stitching framework to realize spatial stitching and temporal stabilization with unsupervised learning simultaneously. First, different from existing learning-based image stitching solutions that typically warp one image to align with another, we suppose a virtual midplane between original image planes and project them onto it. Concretely, we design a differentiable bidirectional decomposition module to disentangle the homography transformation and incorporate it into our spatial warp, evenly spreading alignment burdens and projective distortions across two views. Then, inspired by camera paths in video stabilization, we derive the mathematical expression of stitching trajectories in video stitching by elaborately integrating spatial and temporal warps. Finally, a warp smoothing model is presented to produce stable stitched videos with a hybrid loss to simultaneously encourage content alignment, trajectory smoothness, and online collaboration. Compared with StabStitch that sacrifices alignment for stabilization, StabStitch++ makes no compromise and optimizes both of them simultaneously, especially in the online mode. To establish an evaluation benchmark and train the learning framework, we build a video stitching dataset with a rich diversity in camera motions and scenes. Experiments exhibit that StabStitch++ surpasses current solutions in stitching performance, robustness, and efficiency, offering compelling advancements in this field by building a real-time online video stitching system.

Figures (9)

• Figure 1: The occurrence and elimination of warping shakes. Left: stable camera trajectories for input videos. Middle: warping shakes are produced by image stitching, yielding unsmooth stitching trajectories. Right: the proposed StabStitch++ eliminates these shakes successfully.
• Figure 2: The overview of StabStitch++. We first get the spatial and temporal meshes through spatial and temporal warp models. Then stitching trajectories can be derived by integrating spatial and temporal warps. Finally, a warp smoothing model is leveraged to produce stable stitched frames.
• Figure 3: The network structures of our warping models. The spatial warp, temporal warp, and warp smoothing models are depicted in (a)(b)(d).
• Figure 4: The online stitching mode. We define a sliding window to process a short sequence and display the last frame on the online screen.
• Figure 5: The proposed StabStitch-D dataset. Left: several video examples from diverse scenes. Right: the distribution of video duration time.