WeatherQA: Can Multimodal Language Models Reason about Severe Weather?

TL;DR

WeatherQA tackles the problem of reasoning about severe weather by introducing a first multimodal dataset that pairs imagery of ingredient-based forecasting parameters, surface observations, and radar reflectivity with expert-written Mesoscale Discussions. The authors define two challenging tasks, Areas Affected and Concerning, and benchmark state-of-the-art vision-language models, exposing a substantial gap to human meteorologists. A three-stage training approach for a fine-tuned VLM is described, and an extensive case study with meteorologists provides insights into model strengths and failure modes. The work highlights the need for domain-aligned training, better data integration, and expert oversight to realize reliable AI-assisted weather forecasting with real-world impact.

Abstract

Severe convective weather events, such as hail, tornadoes, and thunderstorms, often occur quickly yet cause significant damage, costing billions of dollars every year. This highlights the importance of forecasting severe weather threats hours in advance to better prepare meteorologists and residents in at-risk areas. Can modern large foundation models perform such forecasting? Existing weather benchmarks typically focus only on predicting time-series changes in certain weather parameters (e.g., temperature, moisture) with text-only features. In this work, we introduce WeatherQA, the first multimodal dataset designed for machines to reason about complex combinations of weather parameters (a.k.a., ingredients) and predict severe weather in real-world scenarios. The dataset includes over 8,000 (multi-images, text) pairs for diverse severe weather events. Each pair contains rich information crucial for forecasting -- the images describe the ingredients capturing environmental instability, surface observations, and radar reflectivity, and the text contains forecast analyses written by human experts. With WeatherQA, we evaluate state-of-the-art vision language models, including GPT4, Claude3.5, Gemini-1.5, and a fine-tuned Llama3-based VLM, by designing two challenging tasks: (1) multi-choice QA for predicting affected area and (2) classification of the development potential of severe convection. These tasks require deep understanding of domain knowledge (e.g., atmospheric dynamics) and complex reasoning over multimodal data (e.g., interactions between weather parameters). We show a substantial gap between the strongest VLM, GPT4o, and human reasoning. Our comprehensive case study with meteorologists further reveals the weaknesses of the models, suggesting that better training and data integration are necessary to bridge this gap. WeatherQA link: https://github.com/chengqianma/WeatherQA.

Figures (9)

• Figure 1: An Overview of WeatherQA. (A): The data curation process involves selecting ingredient-based parameters by forecasters and pairing them with the date and hour from the Mesoscale Discussion to extract the corresponding Mesoscale Analysis. (B): We introduce two downstream tasks which are identifying regions with potential severe weather threats and identifying the types of severe weather concerns issued in a Mesoscale Discussion. A case study investigates the VLM-generated output of Mesoscale Discussions reviewed by experts.
• Figure 2: Comparison of VLMs' output generated from current timestamp and weather condition images (input) to simplified mesoscale discussions written by experts (ground truth).
• Figure 3: An overview of Fine-tuned VLM architecture
• Figure 4: Distribution of Expert Ratings for 3-shot GPT-4o and Fine-tuned-VLM (Llama2)
• Figure 5: Similar to Figure \ref{['fig:case study demo']} but for case 2020 MCD 0061