Harnessing Meta-Learning for Improving Full-Frame Video Stabilization

TL;DR

The paper addresses the challenge of robust full-frame video stabilization with end-to-end pixel-level synthesis by introducing scene-adaptive meta-learning for rapid test-time adaption. It formulates a two-loop MAML-inspired framework that adapts stabilization models to short frame sequences using inner-loop losses focused on stability and perceptual quality, while outer-loop losses guide behavior toward realistic, high-quality stabilization with ground-truth-like targets. The approach leverages a rigid-affine transform regression module and combines perceptual, contextual, and Gram-based penalties to preserve content during adaptation. Empirical results show consistent gains in stability and quality across baselines, with DIFRINT achieving state-of-the-art performance after limited adaptation, demonstrating the practical impact of fast, task-aware adaptation for real-world video stabilization tasks.

Abstract

Video stabilization is a longstanding computer vision problem, particularly pixel-level synthesis solutions for video stabilization which synthesize full frames add to the complexity of this task. These techniques aim to stabilize videos by synthesizing full frames while enhancing the stability of the considered video. This intensifies the complexity of the task due to the distinct mix of unique motion profiles and visual content present in each video sequence, making robust generalization with fixed parameters difficult. In our study, we introduce a novel approach to enhance the performance of pixel-level synthesis solutions for video stabilization by adapting these models to individual input video sequences. The proposed adaptation exploits low-level visual cues accessible during test-time to improve both the stability and quality of resulting videos. We highlight the efficacy of our methodology of "test-time adaptation" through simple fine-tuning of one of these models, followed by significant stability gain via the integration of meta-learning techniques. Notably, significant improvement is achieved with only a single adaptation step. The versatility of the proposed algorithm is demonstrated by consistently improving the performance of various pixel-level synthesis models for video stabilization in real-world scenarios.

Figures (6)

• Figure 1: Recurrence related artifacts. Wobble artifacts observed in the frame recurrent settings for full-frame video stabilization models. Please note that this figure includes animated content and is best viewed on a computer with Adobe PDF Reader.
• Figure 2: Overview of the proposed meta-training process. This figure illustrates the overall pipeline of the training process. The model in the inner loop gets a sequence of local temporal windows ($S_t \in \mathcal{D_{T}}$) and synthesizes stable frames. The synthesized frames are penalized according to the aligned frames in the inner loop. For the outer loop, the deviation of synthesized frames is measured with the corresponding DeepStab wang2018deep stable frames. At inference time, only the inner loop optimization is needed.
• Figure 3: Affine alignment. This affine alignment strategy is analogous to the classical stabilization strategies which estimate and smooth transforms to stabilize videos. Please note that these frames are not neighboring frames and were selected to highlight the crops near the image boundaries in aligned frames $\tilde{V}$.
• Figure 4: Contribution of each objective function. a) The effects of stability loss during the adaptation stage. A higher weight for the proposed stability loss positively affects the stability score. b) The effects of quality loss during the adaptation stage. A higher weight for quality loss positively affects the distortion score.
• Figure 5: Finetuning vs meta-inference. A comparison of the finetuned and the meta-trained models highlights that it takes significant finetuning iterations for a minuscule improvement. Whereas, the proposed algorithm allows for a significant improvement with a single adaptation pass over the video sequence.