Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TranStable: Towards Robust Pixel-level Online Video Stabilization by Jointing Transformer and CNN

zhizhen li, tianyi zhuo, Yifei Cao, Jizhe Yu, Yu Liu

TL;DR

TranStable tackles online video stabilization by fusing global and local cues through a Transformer-CNN generator (TUNet) and enforcing pixel-level realism with a Swin Transformer–based Stability Discriminator Module (SDM). The Hierarchical Adaptive Fusion Module (HAFM) enables effective cross-branch interaction, producing robust pixel-level warping maps that minimize jitter while preserving texture and field of view. Training combines content, shape, and temporal losses, along with discriminator-based guidance, and uses a 31-frame sliding window for inference to enforce temporal consistency. On NUS, DeepStab, and Selfie benchmarks, TranStable achieves state-of-the-art performance with improved cropping, reduced distortion, and smoother stabilization, offering an efficient online solution with minimal post-processing.

Abstract

Video stabilization often struggles with distortion and excessive cropping. This paper proposes a novel end-to-end framework, named TranStable, to address these challenges, comprising a genera tor and a discriminator. We establish TransformerUNet (TUNet) as the generator to utilize the Hierarchical Adaptive Fusion Module (HAFM), integrating Transformer and CNN to leverage both global and local features across multiple visual cues. By modeling frame-wise relationships, it generates robust pixel-level warping maps for stable geometric transformations. Furthermore, we design the Stability Discriminator Module (SDM), which provides pixel-wise supervision for authenticity and consistency in training period, ensuring more complete field-of-view while minimizing jitter artifacts and enhancing visual fidelity. Extensive experiments on NUS, DeepStab, and Selfie benchmarks demonstrate state-of-the-art performance.

TranStable: Towards Robust Pixel-level Online Video Stabilization by Jointing Transformer and CNN

TL;DR

TranStable tackles online video stabilization by fusing global and local cues through a Transformer-CNN generator (TUNet) and enforcing pixel-level realism with a Swin Transformer–based Stability Discriminator Module (SDM). The Hierarchical Adaptive Fusion Module (HAFM) enables effective cross-branch interaction, producing robust pixel-level warping maps that minimize jitter while preserving texture and field of view. Training combines content, shape, and temporal losses, along with discriminator-based guidance, and uses a 31-frame sliding window for inference to enforce temporal consistency. On NUS, DeepStab, and Selfie benchmarks, TranStable achieves state-of-the-art performance with improved cropping, reduced distortion, and smoother stabilization, offering an efficient online solution with minimal post-processing.

Abstract

Video stabilization often struggles with distortion and excessive cropping. This paper proposes a novel end-to-end framework, named TranStable, to address these challenges, comprising a genera tor and a discriminator. We establish TransformerUNet (TUNet) as the generator to utilize the Hierarchical Adaptive Fusion Module (HAFM), integrating Transformer and CNN to leverage both global and local features across multiple visual cues. By modeling frame-wise relationships, it generates robust pixel-level warping maps for stable geometric transformations. Furthermore, we design the Stability Discriminator Module (SDM), which provides pixel-wise supervision for authenticity and consistency in training period, ensuring more complete field-of-view while minimizing jitter artifacts and enhancing visual fidelity. Extensive experiments on NUS, DeepStab, and Selfie benchmarks demonstrate state-of-the-art performance.
Paper Structure (22 sections, 21 equations, 6 figures, 3 tables, 2 algorithms)

This paper contains 22 sections, 21 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Video Stabilization
  4. Video Stabilization based on Deep Learning
  5. The Proposed Method
  6. The Overview Pipeline
  7. Transformer-UNet(TUNet)
  8. Stability Discriminator Module(SDM)
  9. Training
  10. Data Pre-processing
  11. Loss Function Design
  12. Post-processing and Inference Stage
  13. Experiments
  14. Implementation Details
  15. Evaluation on datasets
  16. ...and 7 more sections

Figures (6)

  • Figure 1: A comparison of our approach with state-of-the-art stabilizers. Across various scenarios, our method exhibits a more comprehensive field of view and enhanced visual fidelity. The locally magnified area within the red box highlights the strengths of our model.
  • Figure 2: Overall of our approach. (a) shows us the pipeline consists of two components: a generator and a discriminator. The generator processes consecutive video sequences to generate horizontal and vertical warping maps, which are applied to unstable frames to produce stabilized frames. The discriminator calculates the loss between the estimated frames and the ground truth frames, gradually improving the ability to generate stabilized frames. The generator utilizes a TUNet, as shown in (b) and (c), which depict the TUNet structure and highlight the HFAM component that bridges the Transformer and CNN. (d) illustrates the SDM structure, designed based on the Swin-Transformer. The seamless integration of the generator and discriminator enables the stabilization functionality, with only the generator being used during the inference stage.
  • Figure 3: The relationship between two neighboring points of a point in the grid is as follows: (a) Two adjacent points are positioned on opposite sides of the center point; (b) Two adjacent points are located on the same side of the center point.
  • Figure 4: A detailed comparison of the output frames generated using different discriminators is provided.
  • Figure 5: Quantitative analysis. We take comparison of the output frames of certain video generated by diverse stabilizers.
  • ...and 1 more figures