LangPrecip: Language-Aware Multimodal Precipitation Nowcasting

TL;DR

LangPrecip reframes short-term precipitation nowcasting as semantically constrained trajectory generation in latent space, leveraging Rectified Flow to fuse radar dynamics with meteorological text. The system encodes radar sequences with a VAE and grounds future motion with text descriptions embedded by a T5 encoder, conditioning on c_ctx = {X_{0:4}, m} to steer plausible trajectories without sacrificing uncertainty. A novel WCUB decoder combines data-consistency with wavelet priors, producing high-resolution forecasts that preserve spatial structure while recovering details. On Swedish SWish and MRMS benchmarks, LangPrecip achieves substantial gains in heavy-precipitation detection (over 60% on Swedish and ~19% on MRMS at 80 minutes) and strong event-level metrics, demonstrating the practical value of language-guided, multimodal nowcasting. Limitations include reliance on motion descriptions quality and coarse-grained semantics, motivating future work on expert-in-the-loop annotations and richer motion representations.

Abstract

Short-term precipitation nowcasting is an inherently uncertain and under-constrained spatiotemporal forecasting problem, especially for rapidly evolving and extreme weather events. Existing generative approaches rely primarily on visual conditioning, leaving future motion weakly constrained and ambiguous. We propose a language-aware multimodal nowcasting framework(LangPrecip) that treats meteorological text as a semantic motion constraint on precipitation evolution. By formulating nowcasting as a semantically constrained trajectory generation problem under the Rectified Flow paradigm, our method enables efficient and physically consistent integration of textual and radar information in latent space.We further introduce LangPrecip-160k, a large-scale multimodal dataset with 160k paired radar sequences and motion descriptions. Experiments on Swedish and MRMS datasets show consistent improvements over state-of-the-art methods, achieving over 60 \% and 19\% gains in heavy-rainfall CSI at an 80-minute lead time.

Figures (9)

• Figure 1: Overview of the proposed LangPrecip framework. Radar echo sequences are first encoded into a latent space, where spatiotemporal self-attention captures visual dynamics while semantic motion descriptions provide high-level constraints on precipitation evolution. The two streams are integrated in the latent space to guide future trajectory generation. Leveraging the LangPrecip-160K text–radar paired dataset, the model produces temporally coherent forecasts and achieves substantial performance gains, particularly for heavy precipitation events at long lead times.
• Figure 2: LangPrecip-160K dataset construction pipeline.
• Figure 3: The decoder uses a dual-path design that combines UpBlocks with Temporal Shift Modules for cross-frame alignment and wavelet-based unfolding blocks for data-consistent refinement, followed by feature fusion to produce the final output.
• Figure 4: The Wavelet Consistency Unfolding Block (WCUB) consists of a data-consistency module for gradient updates and a wavelet-prior module that applies soft shrinkage to high-frequency coefficients for detail restoration.
• Figure 5: Spatial precipitation forecasts at multiple lead times (5-80 minutes) for different models with corresponding skill scores (CSI $\ge$ 0.06 mm/h, CSI $\ge$ 6.3 mm/h) during a Swedish weather event in Central Sweden (centered approximately at 63.17°N, 14.63°E, in Östersund) on October 3, 2021, at 16:45.