Omni-Weather: Unified Multimodal Foundation Model for Weather Generation and Understanding

TL;DR

Omni-Weather introduces a unified multimodal framework that simultaneously handles weather generation and weather understanding within a single backbone, addressing both predictive accuracy and interpretability. The model leverages a shared Transformer backbone with task-conditioned prompts and modality-specific encoders, enabling nowcasting, radar inversion, and diagnostic reasoning in one system. A Chain-of-Thought dataset and integration strategy provide explicit causal reasoning to improve perceptual fidelity and explainability, with a joint training objective that benefits both generation and understanding. Experimental results show state-of-the-art performance across tasks and demonstrate mutual gains from joint training and reasoning, underscoring the potential of unified weather intelligence for operational forecasting and risk assessment.

Abstract

Weather modeling requires both accurate prediction and mechanistic interpretation, yet existing methods treat these goals in isolation, separating generation from understanding. To address this gap, we present Omni-Weather, the first multimodal foundation model that unifies weather generation and understanding within a single architecture. Omni-Weather integrates a radar encoder for weather generation tasks, followed by unified processing using a shared self-attention mechanism. Moreover, we construct a Chain-of-Thought dataset for causal reasoning in weather generation, enabling interpretable outputs and improved perceptual quality. Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding. Our findings further indicate that generative and understanding tasks in the weather domain can mutually enhance each other. Omni-Weather also demonstrates the feasibility and value of unifying weather generation and understanding.

Figures (12)

• Figure 1: Illustration of Omni-Weather’s unified capabilities.
• Figure 2: Comparison between separated architectures for weather understanding / generation (top) and unified framework with shared self-attention (bottom).
• Figure 3: Framework and Task paradigm of Omni-Weather.
• Figure 4: Construction of our CoT data. First, we preprocess the raw SEVIR data to obtain high-quality input / output frame pairs. Second, we carefully design prompts and leverage GPT-4o for attributes annotation. Third, the annotated attributes are incorporated into CoT prompts to generate CoT annotations, followed by a quality verification step to produce the final CoT dataset.
• Figure 5: A set of qualitative results. We show two radar inversion examples with think traces, a nowcasting case where Omni-Weather (with think trace) is compared against CasCast, DiffCast, and EarthFormer, and one example each of radar image and sequence understanding with attribute scores and textual evaluations. Omni-Weather surpasses all baselines.