Quantizing Space and Time: Fusing Time Series and Images for Earth Observation

TL;DR

This work addresses the challenge of fusing time-series and single-timestamp images in a task-agnostic, token-based framework. By quantizing both modalities into discrete tokens and aligning them via masked correlation learning, the approach enables bidirectional generation (images from time series and vice versa) and robust pretraining for diverse Earth observation tasks. Empirical results show improved crop-yield predictions and the capability to generate consistent global temperature profiles from satellite imagery, along with counterfactual analyses and gradient-based robustness insights. The framework offers scalable, interpretable multimodal representations with practical impact for climate science, agriculture, and beyond, and plans to release code, data, and pretrained weights.

Abstract

We propose a task-agnostic framework for multimodal fusion of time series and single timestamp images, enabling cross-modal generation and robust downstream performance. Our approach explores deterministic and learned strategies for time series quantization and then leverages a masked correlation learning objective, aligning discrete image and time series tokens in a unified representation space. Instantiated in the Earth observation domain, the pretrained model generates consistent global temperature profiles from satellite imagery and is validated through counterfactual experiments. Across downstream tasks, our task-agnostic pretraining outperforms task-specific fusion by 6% in R^2 and 2% in RMSE on average, and exceeds baseline methods by 50% in R^2 and 12% in RMSE. Finally, we analyze gradient sensitivity across modalities, providing insights into model robustness. Code, data, and weights will be released under a permissive license.

Figures (18)

• Figure 1: Framework for task-agnostic fusion of quantized time series and image tokens. Left: Schematic comparison of tokenization strategies for uni-variate time series. (a) Deterministic: Uniform value‑range partitions. (b) Deterministic: Quantile‑based partitions for balanced codebook usage. (c) Learned: Encoder–latent finite scalar quantization (FSQ) producing discrete tokens. In addition: Quantization of image patches via learned FSQ. Right: Cross-modal alignment learning leverages quantized tokens from images and timeseries to correlate the modalities.
• Figure 2: High-level comparison of task-specific supervised fusion of time series and images and task-agnostic self-supervised fusion (ours).
• Figure 3: Comparison of interpolation strategies for temperature (left) and precipitation (right). Nearest-neighbor interpolation preserves original grid values but introduces blocky artifacts, while linear interpolation produces smoother fields.
• Figure 4: Downstream task approach for crop yield prediction: Images are encoded individually, followed by modality-wise mean or global mean pooling. Next, spatio-temporal mean pooling merges produced tokens into a single embedding per county. A MLP head predicts overall crop production.
• Figure 5: Comparison of distribution of true and generated daily surface temperatures across minimum, mean, and maximum temperature profiles.