Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatio-Temporal Turbulence Mitigation: A Translational Perspective

Xingguang Zhang, Nicholas Chimitt, Yiheng Chi, Zhiyuan Mao, Stanley H. Chan

TL;DR

Atmospheric turbulence distorts long-range images via tilt and blur, posing a challenging inverse problem. The paper introduces DATUM, a recurrent deep-network that embeds classical turbulence mitigation biases into an end-to-end framework, leveraging feature-to-reference registration, temporal fusion, and decoupled tilt/blur processing. It also presents ATSyn, a physics-based, large-scale dataset generated with a Zernike-based turbulence model to improve real-world generalization. Experiments show DATUM achieves state-of-the-art restoration quality with up to 10x faster processing and strong generalization on dynamic and static turbulence scenarios, including real-world videos and downstream recognition tasks.

Abstract

Recovering images distorted by atmospheric turbulence is a challenging inverse problem due to the stochastic nature of turbulence. Although numerous turbulence mitigation (TM) algorithms have been proposed, their efficiency and generalization to real-world dynamic scenarios remain severely limited. Building upon the intuitions of classical TM algorithms, we present the Deep Atmospheric TUrbulence Mitigation network (DATUM). DATUM aims to overcome major challenges when transitioning from classical to deep learning approaches. By carefully integrating the merits of classical multi-frame TM methods into a deep network structure, we demonstrate that DATUM can efficiently perform long-range temporal aggregation using a recurrent fashion, while deformable attention and temporal-channel attention seamlessly facilitate pixel registration and lucky imaging. With additional supervision, tilt and blur degradation can be jointly mitigated. These inductive biases empower DATUM to significantly outperform existing methods while delivering a tenfold increase in processing speed. A large-scale training dataset, ATSyn, is presented as a co-invention to enable generalization in real turbulence. Our code and datasets are available at https://xg416.github.io/DATUM.

Spatio-Temporal Turbulence Mitigation: A Translational Perspective

TL;DR

Atmospheric turbulence distorts long-range images via tilt and blur, posing a challenging inverse problem. The paper introduces DATUM, a recurrent deep-network that embeds classical turbulence mitigation biases into an end-to-end framework, leveraging feature-to-reference registration, temporal fusion, and decoupled tilt/blur processing. It also presents ATSyn, a physics-based, large-scale dataset generated with a Zernike-based turbulence model to improve real-world generalization. Experiments show DATUM achieves state-of-the-art restoration quality with up to 10x faster processing and strong generalization on dynamic and static turbulence scenarios, including real-world videos and downstream recognition tasks.

Abstract

Recovering images distorted by atmospheric turbulence is a challenging inverse problem due to the stochastic nature of turbulence. Although numerous turbulence mitigation (TM) algorithms have been proposed, their efficiency and generalization to real-world dynamic scenarios remain severely limited. Building upon the intuitions of classical TM algorithms, we present the Deep Atmospheric TUrbulence Mitigation network (DATUM). DATUM aims to overcome major challenges when transitioning from classical to deep learning approaches. By carefully integrating the merits of classical multi-frame TM methods into a deep network structure, we demonstrate that DATUM can efficiently perform long-range temporal aggregation using a recurrent fashion, while deformable attention and temporal-channel attention seamlessly facilitate pixel registration and lucky imaging. With additional supervision, tilt and blur degradation can be jointly mitigated. These inductive biases empower DATUM to significantly outperform existing methods while delivering a tenfold increase in processing speed. A large-scale training dataset, ATSyn, is presented as a co-invention to enable generalization in real turbulence. Our code and datasets are available at https://xg416.github.io/DATUM.
Paper Structure (33 sections, 20 equations, 13 figures, 7 tables, 1 algorithm)

This paper contains 33 sections, 20 equations, 13 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Turbulence modeling
  4. Conventional turbulence mitigation
  5. Learning-based turbulence mitigation
  6. Proposed method
  7. Insights from Classical Methods
  8. DATUM network
  9. Overview
  10. Components
  11. ATSyn dataset
  12. Physics-based data synthesis
  13. Guideline of implementation
  14. Experiments
  15. Training setting
  16. ...and 18 more sections

Figures (13)

  • Figure 1: Benchmarking video restoration models for turbulence mitigation on our ATSyn-dynamic dataset. The circles in orange are other video-based TM networks, and the circles in blue are representative video deblurring and general restoration networks. The proposed Deep Atmospheric TUrbulence Mitigation network (DATUM) is state-of-the-art while highly efficient.
  • Figure 2: The proposed DATUM network. In this figure, block (a) shows the three common stages proposed by classical TM methods. The corresponding stages in DATUM are shown in block (b), which illustrates the forward time process of the $t\text{-th}$ frame. The dashed line means the information passing from other temporal directions and frames. Block (c), (d), and (e) demonstrate the DAAB, GRUM, and MTCSA modules, respectively, where the input features are marked by green, and the output features are marked by red.
  • Figure 3: Scheme of our data synthesis method.
  • Figure 4: Influence of the number of input frames.
  • Figure 5: Comparison on the real-world turbulence-text dataset. The metric is the average text recognition accuracy of CRNN, DAN, and ASTER tested on the restored images.
  • ...and 8 more figures