Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence

TL;DR

Atmospheric turbulence degrades long-range dynamic video by causing geometric distortions and blur. This work proposes Turb-Seg-Res, a segment-then-restore pipeline that first performs unsupervised motion segmentation via mean optical flow to separate static background from dynamic foreground, then enhances the background and sharpens the entire frame with a transformer trained on a novel tilt-and-blur turbulence simulator. A procedurally generated turbulence video simulator based on 3D simplex and Perlin noise enables rapid, scalable training data creation for the Restormer-based restoration. The method achieves improved geometry recovery and high-frequency detail with competitive latency (≈5.71 s per 1080p frame on A100) across CLEAR, OTIS, and URG-T benchmarks, and the authors release code, simulator, and data to accelerate research in turbulence-robust video restoration.

Abstract

Tackling image degradation due to atmospheric turbulence, particularly in dynamic environment, remains a challenge for long-range imaging systems. Existing techniques have been primarily designed for static scenes or scenes with small motion. This paper presents the first segment-then-restore pipeline for restoring the videos of dynamic scenes in turbulent environment. We leverage mean optical flow with an unsupervised motion segmentation method to separate dynamic and static scene components prior to restoration. After camera shake compensation and segmentation, we introduce foreground/background enhancement leveraging the statistics of turbulence strength and a transformer model trained on a novel noise-based procedural turbulence generator for fast dataset augmentation. Benchmarked against existing restoration methods, our approach restores most of the geometric distortion and enhances sharpness for videos. We make our code, simulator, and data publicly available to advance the field of video restoration from turbulence: riponcs.github.io/TurbSegRes

Figures (8)

• Figure 1: Our pipeline processes dynamic video frames by first stabilizing image sequences using normalized cross-correlation, followed by segmenting moving objects using average optical flow. The background is processed with adaptive filtering and blended with a separately extracted foreground. The background and segmented foreground are seamlessly merged using Poisson pyramid blending. Finally, a transformer architecture, trained on our simulator, refines the combined images.
• Figure 2: Visual representation of motion segmentation across consecutive frames in a video sequence. Our proposed method effectively separates moving subjects (students walking) visualized by the golden outlines.
• Figure 3: Comparison of the original image, noise-added image, and the image with simulated turbulence.
• Figure 4: Comparative visualization of turbulence mitigation methods: AT-Net and Turbnet introduce artifacts, failing to clearly define edges of the road, while TCI maintains edge integrity but blurs moving objects. Our method shows minimal distortion with clear depiction of both static and dynamic elements.
• Figure 5: Analysis of the OTIS dataset reveals distinct outcomes. The upper row, featuring 'Pattern16' (300 frames, $135\times135$ pixels), demonstrates notable clarity in our method compared to AT-Net and TurbNet, which exhibit significant distortion in finer details. The lower row, showcasing 'Fixed Background (Door)' (300 frames, $520\times520$ pixels), highlights our approach's superiority in maintaining geometric integrity, especially in fence structures and the brickwork of stairs and doors. While TCI performs well on the fence, it distorts the circular patterns. Our method excels in preserving straight lines and enhancing edge definition in circular patterns, offering the clearest and least distorted results.