Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Satellite and Weather Data for Crop Type Mapping: An Inverse Modelling Approach

Praveen Ravirathinam, Rahul Ghosh, Ankush Khandelwal, Xiaowei Jia, David Mulla, Vipin Kumar

TL;DR

This work addresses the problem of pixel-level crop-type mapping by leveraging weather information alongside satellite imagery. It introduces WSTATT, an inverse-modeling framework with dual encoders, a spatio-temporal attention mechanism, and a decoder that fuses Daymet weather data with Sentinel-2 imagery to produce accurate crop maps. WSTATT demonstrates improvements over a satellite-only baseline across year-end and early-prediction tasks and reveals discriminative temporal windows guided by weather signals aligned with crop phenology. The approach enables earlier, more reliable crop maps, offering practical benefits for yield projection, resource management, and food security, while highlighting the role of weather in shaping crop dynamics within remote sensing. Key contributions include: (i) formulating crop mapping as an inverse problem that jointly uses weather and satellite data; (ii) achieving superior year-end and early-prediction performance over STATT; (iii) providing attention-based insights into discriminative time windows and the weather–phenology relationship; and (iv) releasing reproducible code for multimodal, weather-aware crop mapping.

Abstract

Accurate and timely crop mapping is essential for yield estimation, insurance claims, and conservation efforts. Over the years, many successful machine learning models for crop mapping have been developed that use just the multi-spectral imagery from satellites to predict crop type over the area of interest. However, these traditional methods do not account for the physical processes that govern crop growth. At a high level, crop growth can be envisioned as physical parameters, such as weather and soil type, acting upon the plant leading to crop growth which can be observed via satellites. In this paper, we propose Weather-based Spatio-Temporal segmentation network with ATTention (WSTATT), a deep learning model that leverages this understanding of crop growth by formulating it as an inverse model that combines weather (Daymet) and satellite imagery (Sentinel-2) to generate accurate crop maps. We show that our approach provides significant improvements over existing algorithms that solely rely on spectral imagery by comparing segmentation maps and F1 classification scores. Furthermore, effective use of attention in WSTATT architecture enables detection of crop types earlier in the season (up to 5 months in advance), which is very useful for improving food supply projections. We finally discuss the impact of weather by correlating our results with crop phenology to show that WSTATT is able to capture physical properties of crop growth.

Combining Satellite and Weather Data for Crop Type Mapping: An Inverse Modelling Approach

TL;DR

This work addresses the problem of pixel-level crop-type mapping by leveraging weather information alongside satellite imagery. It introduces WSTATT, an inverse-modeling framework with dual encoders, a spatio-temporal attention mechanism, and a decoder that fuses Daymet weather data with Sentinel-2 imagery to produce accurate crop maps. WSTATT demonstrates improvements over a satellite-only baseline across year-end and early-prediction tasks and reveals discriminative temporal windows guided by weather signals aligned with crop phenology. The approach enables earlier, more reliable crop maps, offering practical benefits for yield projection, resource management, and food security, while highlighting the role of weather in shaping crop dynamics within remote sensing. Key contributions include: (i) formulating crop mapping as an inverse problem that jointly uses weather and satellite data; (ii) achieving superior year-end and early-prediction performance over STATT; (iii) providing attention-based insights into discriminative time windows and the weather–phenology relationship; and (iv) releasing reproducible code for multimodal, weather-aware crop mapping.

Abstract

Accurate and timely crop mapping is essential for yield estimation, insurance claims, and conservation efforts. Over the years, many successful machine learning models for crop mapping have been developed that use just the multi-spectral imagery from satellites to predict crop type over the area of interest. However, these traditional methods do not account for the physical processes that govern crop growth. At a high level, crop growth can be envisioned as physical parameters, such as weather and soil type, acting upon the plant leading to crop growth which can be observed via satellites. In this paper, we propose Weather-based Spatio-Temporal segmentation network with ATTention (WSTATT), a deep learning model that leverages this understanding of crop growth by formulating it as an inverse model that combines weather (Daymet) and satellite imagery (Sentinel-2) to generate accurate crop maps. We show that our approach provides significant improvements over existing algorithms that solely rely on spectral imagery by comparing segmentation maps and F1 classification scores. Furthermore, effective use of attention in WSTATT architecture enables detection of crop types earlier in the season (up to 5 months in advance), which is very useful for improving food supply projections. We finally discuss the impact of weather by correlating our results with crop phenology to show that WSTATT is able to capture physical properties of crop growth.
Paper Structure (20 sections, 1 equation, 6 figures, 2 tables)

This paper contains 20 sections, 1 equation, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Land Cover Mapping
  4. Weather based Remote Sensing
  5. Method
  6. Problem Setting
  7. Deep Learning model
  8. Dataset and Experimental Setup
  9. Study Region
  10. Satellite Imagery
  11. Weather Data
  12. Labels
  13. Implementation details
  14. Results
  15. Year End Prediction
  16. ...and 5 more sections

Figures (6)

  • Figure 1: A diagrammatic representation of the physics point of view understanding of crop growth
  • Figure 2: A diagrammatic representation of our proposed deep learning architecture WSTATT (Weather based Spatio-Temporal segmentation networks with ATTention)
  • Figure 3: Visualisation of T11SKA Sentinel Tile CDL labels and grid split used in our experiments. Regions in Purple represent Unknown and were not used in analysis
  • Figure 4: Cases where weather based model (WSTATT) gets a good prediction well before the end of the year
  • Figure 5: Cases where as more data is provided the prediction of incorrect fields progressively gets better (Please look at fields denoted by arrow marks)
  • ...and 1 more figures