Multi-modal learning for geospatial vegetation forecasting

TL;DR

This work presents the first models for continental-scale vegetation modeling at fine resolution able to capture anomalies beyond the seasonal cycle, thereby paving the way for predicting vegetation health and behaviour in response to climate variability and extremes.

Abstract

The innovative application of precise geospatial vegetation forecasting holds immense potential across diverse sectors, including agriculture, forestry, humanitarian aid, and carbon accounting. To leverage the vast availability of satellite imagery for this task, various works have applied deep neural networks for predicting multispectral images in photorealistic quality. However, the important area of vegetation dynamics has not been thoroughly explored. Our study breaks new ground by introducing GreenEarthNet, the first dataset specifically designed for high-resolution vegetation forecasting, and Contextformer, a novel deep learning approach for predicting vegetation greenness from Sentinel 2 satellite images with fine resolution across Europe. Our multi-modal transformer model Contextformer leverages spatial context through a vision backbone and predicts the temporal dynamics on local context patches incorporating meteorological time series in a parameter-efficient manner. The GreenEarthNet dataset features a learned cloud mask and an appropriate evaluation scheme for vegetation modeling. It also maintains compatibility with the existing satellite imagery forecasting dataset EarthNet2021, enabling cross-dataset model comparisons. Our extensive qualitative and quantitative analyses reveal that our methods outperform a broad range of baseline techniques. This includes surpassing previous state-of-the-art models on EarthNet2021, as well as adapted models from time series forecasting and video prediction. To the best of our knowledge, this work presents the first models for continental-scale vegetation modeling at fine resolution able to capture anomalies beyond the seasonal cycle, thereby paving the way for predicting vegetation health and behaviour in response to climate variability and extremes.

Figures (10)

• Figure 1: Future vegetation status $\hat{V}$ is predicted with deep learning models $f$ from past satellite imagery $X$, past and future weather $C$ and elevation $E$. The dataset GreenEarthNet spans across Europe with minicubes split into train (red markers), temporal OOD test (ood-t, yellow) and spatio-temporal OOD test (ood-st, blue) subsets.
• Figure 2: The architecture of our proposed Contextformer.
• Figure 3: Qualitative Results of Contextformer for one OOD-t minicube located near Oradea, Romania. The top-left shows timeseries for all pixels (mean and std. dev.) and for a single pixel (green square on top right). The right side shows image timeseries of cloud-masked target and predicted NDVI alongside their difference.
• Figure 4: Model performance comparing meteo-guided models (blue) with the ablation not using weather (black bar is std. dev. from three random seeds).
• Figure 5: Contextformer model skill for different seasons and landcover on the OOD-t test set.