High-Dynamic Radar Sequence Prediction for Weather Nowcasting Using Spatiotemporal Coherent Gaussian Representation

TL;DR

The paper addresses the challenge of 3D weather nowcasting by introducing SpatioTemporal Coherent Gaussian Splatting (STC-GS) to represent high-resolution 3D radar data as temporally coherent Gaussian groups and a memory-augmented predictor GauMamba to forecast their evolution. The approach combines a bidirectional Gaussian reconstruction scheme with local motion priors and a global density prior, enabling stable, high-fidelity reconstruction of dynamic radar scenes. It also introduces GauMamba, a memory-augmented Mamba network that efficiently handles a large set of Gaussian tokens to predict future radar changes, achieving state-of-the-art performance on NEXRAD and MOSAIC with substantial MAE and CSI improvements. The work demonstrates that re-representing 3D radar data as Gaussians and forecasting their evolution can yield 16× higher spatial resolution and practical, scalable nowcasting capabilities, with potential impact on disaster management and urban planning.

Abstract

Weather nowcasting is an essential task that involves predicting future radar echo sequences based on current observations, offering significant benefits for disaster management, transportation, and urban planning. Current prediction methods are limited by training and storage efficiency, mainly focusing on 2D spatial predictions at specific altitudes. Meanwhile, 3D volumetric predictions at each timestamp remain largely unexplored. To address such a challenge, we introduce a comprehensive framework for 3D radar sequence prediction in weather nowcasting, using the newly proposed SpatioTemporal Coherent Gaussian Splatting (STC-GS) for dynamic radar representation and GauMamba for efficient and accurate forecasting. Specifically, rather than relying on a 4D Gaussian for dynamic scene reconstruction, STC-GS optimizes 3D scenes at each frame by employing a group of Gaussians while effectively capturing their movements across consecutive frames. It ensures consistent tracking of each Gaussian over time, making it particularly effective for prediction tasks. With the temporally correlated Gaussian groups established, we utilize them to train GauMamba, which integrates a memory mechanism into the Mamba framework. This allows the model to learn the temporal evolution of Gaussian groups while efficiently handling a large volume of Gaussian tokens. As a result, it achieves both efficiency and accuracy in forecasting a wide range of dynamic meteorological radar signals. The experimental results demonstrate that our STC-GS can efficiently represent 3D radar sequences with over $16\times$ higher spatial resolution compared with the existing 3D representation methods, while GauMamba outperforms state-of-the-art methods in forecasting a broad spectrum of high-dynamic weather conditions.

Figures (13)

• Figure 1: Comparisons between 2D and 3D radar sequence prediction frameworks. (a) Current 2D methods mainly focus on 2D spatial predictions at specific altitudes. Limited by training and storage efficiency, it is challenging to extend these methods to 3D architectures directly. (b) The size of the storage occupied by a single frame input is calculated according to the resolution of two datasets. The resolution of MOSAIC is $36 \times 384 \times 512$, 2D input is $384 \times 512$, 3D input is $36 \times 384 \times 512$, and the size of 3D Gaussians is $49,152 \times 11$. The resolution of NEXRAD is $36 \times 512 \times 512$ with 6 channels, the inputs are $6\times512\times512$, $6\times36\times512\times512$ and $49,152 \times 16$.
• Figure 2: Overview of our 3D prediction framework based on STC-GS and GauMamba. (a) STC-GS can effectively compress the size of 3D data while fully representing it. GauMamba is a memory-augmented predictive model that leverages STC-GS for effective and accurate predictions. The STC-GS at Frame $t$ is input into the GauMamba to predict a set of DiffGaussians, $\Delta \mathcal{G}_{t+1}$, representing the differences between $\mathcal{G}_{t+1}$ and $\mathcal{G}_0$. This process is applied iteratively from Frame $0$ to Frame $T_{in} + T_{out} -1$. (b) In the process of radar reconstruction, dual-scale constraints are implemented to capture both the global trends and the local details present in the Gaussian motions.
• Figure 3: The overall architecture of the GauMamba. (a) Main architecture of GauMamba which consists of multiple stacked MambaGRU Block. (b) Detailed architecture design of MambaGRU.
• Figure 4: Qualitative results of reconstruction.
• Figure 5: Memory usage of different methods with various input resolutions.