Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Transformer-Based Time-Series Image Reconstruction for Cloud-Filling Applications

Lujun Li, Yiqun Wang, Radu State

Abstract

Cloud cover in multispectral imagery (MSI) poses significant challenges for early season crop mapping, as it leads to missing or corrupted spectral information. Synthetic aperture radar (SAR) data, which is not affected by cloud interference, offers a complementary solution, but lack sufficient spectral detail for precise crop mapping. To address this, we propose a novel framework, Time-series MSI Image Reconstruction using Vision Transformer (ViT), to reconstruct MSI data in cloud-covered regions by leveraging the temporal coherence of MSI and the complementary information from SAR from the attention mechanism. Comprehensive experiments, using rigorous reconstruction evaluation metrics, demonstrate that Time-series ViT framework significantly outperforms baselines that use non-time-series MSI and SAR or time-series MSI without SAR, effectively enhancing MSI image reconstruction in cloud-covered regions.

Vision Transformer-Based Time-Series Image Reconstruction for Cloud-Filling Applications

Abstract

Cloud cover in multispectral imagery (MSI) poses significant challenges for early season crop mapping, as it leads to missing or corrupted spectral information. Synthetic aperture radar (SAR) data, which is not affected by cloud interference, offers a complementary solution, but lack sufficient spectral detail for precise crop mapping. To address this, we propose a novel framework, Time-series MSI Image Reconstruction using Vision Transformer (ViT), to reconstruct MSI data in cloud-covered regions by leveraging the temporal coherence of MSI and the complementary information from SAR from the attention mechanism. Comprehensive experiments, using rigorous reconstruction evaluation metrics, demonstrate that Time-series ViT framework significantly outperforms baselines that use non-time-series MSI and SAR or time-series MSI without SAR, effectively enhancing MSI image reconstruction in cloud-covered regions.

Paper Structure

This paper contains 15 sections, 2 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Study Areas and Data Preparation
  4. Methodology
  5. Problem Definition and Framework
  6. Problem Definition
  7. General Framework
  8. Convolutional Patch Projection (CPP)
  9. MHSA
  10. Linear Patch Decoder & Loss Function
  11. Experiments
  12. Experiments Setup
  13. Datasets & Settings
  14. Results
  15. Conclusions and Future Works

Figures (3)

  • Figure 1: The Study Area Traill County located in North Dakota, the USA.
  • Figure 2: The proposed Time-Series ViT reconstruction Structure
  • Figure 3: The reconstructed images from the time-series input model are shown, with the x-axis representing the days of the year. "Inputs" refer to the cloud-augmented data, while "Targets" indicate the desired outputs (black pixels are real clouds). The remaining images display the results from the three models trained in this study, with a cloud count of 20 in this example.