NeRT: Implicit Neural Representations for General Unsupervised Turbulence Mitigation

TL;DR

NeRT addresses the challenge of mitigating atmospheric and water turbulence without requiring paired training data. It uses an unsupervised, implicit neural representation guided by a physically grounded tilt-then-blur forward model $\mathbf{I} = [\mathcal{B} \circ \mathcal{T}]\mathbf{J}$, with Zernike-based tilts and shift-varying PSF kernels learned per frame. The approach yields state-of-the-art or competitive results across synthetic and real turbulence datasets, including difficult water ripple and uncontrolled environments, and enables a practical 48× speedup for live video reconstruction after an initialization phase. This work broadens turbulence restoration to general, out-of-domain scenarios and offers a viable path toward real-time, high-fidelity reconstructions in challenging imaging conditions.

Abstract

The atmospheric and water turbulence mitigation problems have emerged as challenging inverse problems in computer vision and optics communities over the years. However, current methods either rely heavily on the quality of the training dataset or fail to generalize over various scenarios, such as static scenes, dynamic scenes, and text reconstructions. We propose a general implicit neural representation for unsupervised atmospheric and water turbulence mitigation (NeRT). NeRT leverages the implicit neural representations and the physically correct tilt-then-blur turbulence model to reconstruct the clean, undistorted image, given only dozens of distorted input images. Moreover, we show that NeRT outperforms the state-of-the-art through various qualitative and quantitative evaluations of atmospheric and water turbulence datasets. Furthermore, we demonstrate the ability of NeRT to eliminate uncontrolled turbulence from real-world environments. Lastly, we incorporate NeRT into continuously captured video sequences and demonstrate $48 \times$ speedup.

Figures (9)

• Figure 1: The overall architecture of NeRT. The network predicts a clean and sharp image $\hat{\mathbf{J}}$, given a series of observed atmospheric turbulence distorted images $\mathbf{I}$. We compute $L_1$ loss between predicted tilted-then-blurred images $\mathbf{I}_{\mathcal{B}\circ \mathcal{T}}$, resampled tilted-then-blurred images $\mathbf{I}^{G}_{\mathcal{B}\circ \mathcal{T}}$, and observed images $\mathbf{I}$ during optimization to update the parameters in image generators $\mathcal{I}_{\phi}$, grid deformers $\mathcal{G}_{\theta}$, and shift-varying blurring $\mathcal{P}_{\alpha}$.
• Figure 2: Simulation of turbulence using $\mathcal{B} \circ \mathcal{T}$, $\mathcal{T} \circ \mathcal{B}$, and the full model, along with their respective error maps with respect to the full model. The error of the blur-then-tilt model, $\mathcal{T} \circ \mathcal{B}$ occurs at edges and high-resolution regions.
• Figure 3: Qualitative results from our newly captured in-the-wild static scene atmospheric turbulence datasets. All sequences are captured using a Sony a7 III mirrorless camera with a 400 mm lens. TurbNet mao2022single and TSR-WGAN jin2021neutralizing are supervised SOTA while CLEAR anantrasirichai2018atmospheric and NDIR li2021unsupervised are unsupervised SOTA. NDIR+Deblur post-processes the NDIR results by applying an off-the-shelf deblurring algorithm li2018learning, which has no domain-specific knowledge of atmospheric turbulence.
• Figure 4: Qualitative results from the static scene atmospheric turbulence datasets, including Siemens star dataset gilles2017open, text dataset mao2022single, door dataset gilles2017open and in-the-wild dataset mao2020image. We compare NeRT with other supervised mao2022single and unsupervised li2021unsupervisedanantrasirichai2018atmospheric SOTA. NeRT is able to achieve high spatial resolution, recover high-contrast text, and reconstruct fine details, such as wire fences. CLEAR anantrasirichai2018atmospheric, TSR-WGAN jin2021neutralizing, and TurbNet mao2022single fails to mitigate the atmospheric turbulence, while NDIR li2021unsupervised fails to preserve fine details of the wire fences and electric poles. NDIR+Deblur relies on an off-the-shelf general-purpose deblurring software li2018learning, which estimates the atmospheric turbulence PSF inaccurately, and produces ringing and noisy artifacts along edges.
• Figure 5: Qualitative results from the dynamic scene atmospheric turbulence dataset, airport dataset anantrasirichai2018atmospheric. CLEAR anantrasirichai2018atmospheric is a single-image reconstruction algorithm while other methods are multi-image reconstruction algorithms. All the input distorted images are fixed the same for all the multi-image reconstruction algorithms. Among those input distorted images, the sharpest one is chosen as the input for the single-image reconstruction algorithm TurbNet zhang2022imaging. We compare NeRT with other supervised mao2022singlejin2021neutralizing and unsupervised li2021unsupervisedanantrasirichai2018atmospheric SOTA. We are able to recover the undistorted and clean logo of "THAI" Airways while other methods show distorted and noisy logos.