DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection

TL;DR

Atmospheric turbulence degrades surveillance video, hindering object detection. DMAT addresses this by jointly training an AT mitigator based on 3D Mamba (SSM) and a transformer-based detector in an end-to-end framework, enabling mutual feature refinement. The approach yields notable gains in detection accuracy (up to $15\%$ mAP across turbulence-corrupted data) and superior restoration quality (PSNR/SSIM) compared with separate restoration or detection pipelines, including real turbulence scenarios. The bidirectional information flow between low-level restoration and high-level detection enhances robustness, particularly for small objects, making the method practically impactful for aerial and surveillance applications.

Abstract

Atmospheric Turbulence (AT) degrades the clarity and accuracy of surveillance imagery, posing challenges not only for visualization quality but also for object classification and scene tracking. Deep learning-based methods have been proposed to improve visual quality, but spatio-temporal distortions remain a significant issue. Although deep learning-based object detection performs well under normal conditions, it struggles to operate effectively on sequences distorted by atmospheric turbulence. In this paper, we propose a novel framework that learns to compensate for distorted features while simultaneously improving visualization and object detection. This end-to-end training strategy leverages and exchanges knowledge of low-level distorted features in the AT mitigator with semantic features extracted in the object detector. Specifically, in the AT mitigator a 3D Mamba-based structure is used to handle the spatio-temporal displacements and blurring caused by turbulence. Optimization is achieved through back-propagation in both the AT mitigator and object detector. Our proposed DMAT outperforms state-of-the-art AT mitigation and object detection systems up to a 15% improvement on datasets corrupted by generated turbulence.

Figures (8)

• Figure 1: Object detection results for all object sizes and classes of our proposed method (DMAT) compared to individual AT-mitigation and object detection methods.
• Figure 2: Diagrams of (a) transitional approach and (b) our proposed approach for joint restoration and object detection under atmospheric turbulence.
• Figure 3: Architecture of the proposed DMAT framework for atmospheric distortion mitigation and object detection.
• Figure 4: AT distortion. (a) Frames 1 and 2. (b) Temporal intensity variation of two pixels. Yellow dots show the benefit of deformable convolutions.
• Figure 5: Object detection performance (mAP[0.50-0.95]) for various object class groupings across all (left) and small (right) object sizes. Number of parameters is the size of the object detector only (for AT and noAT) and the sum of parameters of the AT mitigation model and the object detector.