Zephyrus: An Agentic Framework for Weather Science

TL;DR

Zephyrus tackles the gap between natural language reasoning and high-dimensional weather data by introducing an agentic framework that pairs LLMs with a Python-based execution environment (ZephyrusWorld) and weather-focused tools. It builds two code-generation agents (Direct and Reflective) to solve weather tasks through programmatic data access, forecasting, and climate simulation, evaluated on the ZephyrusBench dataset derived from ERA5 WeatherBench 2. The results show substantial gains over text-only baselines, especially for simpler tasks, and demonstrate the potential of interactive, tool-enabled reasoning in meteorology, while underscoring the difficulty of hard, long-horizon forecasting tasks. The work provides a scalable sandbox for developing agentic workflows in weather science and points to future directions in dataset scale and extended reasoning horizons for improved performance.

Abstract

Foundation models for weather science are pre-trained on vast amounts of structured numerical data and outperform traditional weather forecasting systems. However, these models lack language-based reasoning capabilities, limiting their utility in interactive scientific workflows. Large language models (LLMs) excel at understanding and generating text but cannot reason about high-dimensional meteorological datasets. We bridge this gap by building a novel agentic framework for weather science. Our framework includes a Python code-based environment for agents (ZephyrusWorld) to interact with weather data, featuring tools like an interface to WeatherBench 2 dataset, geoquerying for geographical masks from natural language, weather forecasting, and climate simulation capabilities. We design Zephyrus, a multi-turn LLM-based weather agent that iteratively analyzes weather datasets, observes results, and refines its approach through conversational feedback loops. We accompany the agent with a new benchmark, ZephyrusBench, with a scalable data generation pipeline that constructs diverse question-answer pairs across weather-related tasks, from basic lookups to advanced forecasting, extreme event detection, and counterfactual reasoning. Experiments on this benchmark demonstrate the strong performance of Zephyrus agents over text-only baselines, outperforming them by up to 35 percentage points in correctness. However, on harder tasks, Zephyrus performs similarly to text-only baselines, highlighting the challenging nature of our benchmark and suggesting promising directions for future work.

Figures (10)

• Figure 1: Overview: We develop Zephyrus, an agentic framework for weather science. Given a query, the LLM-based agent Zephyrus writes a code block which is sent to the code execution server. The server orchestrates several tools to execute the code block and returns the execution results to the agent. The agent either decides to execute more code to refine its output or respond back to the user. Refer to Appendix \ref{['sec:model_prompts']} for the full prompt.
• Figure 2: Code Execution Server.Zephyrus sends parallel requests to the server, which distributes them to available workers. Each worker acquires resources from tool pools, loads datasets, injects tools into the execution environment, executes code, and returns results or errors to the agent.
• Figure 3: Semi-synthetic task generation pipeline: Weather-related texts are processed by a claim extraction agent to identify scientifically meaningful observational claims. Verified claims are transformed into reusable templates and manually reviewed. Code is generated by an LLM and verified by humans to validate each sample from a template against ERA5 meteorological data. We combine the verifier code with the templates and WeatherBench data to produce novel samples.
• Figure 4: Percentage of questions in the complete dataset answered correctly by each LLM and model type. Definitions of correctness for each question type are detailed in Appendix \ref{['def_correctness']}.
• Figure 5: Plots showing (top) error distribution on numerical tasks (bottom-left) location accuracy (bottom-right) discussion scores for GPT-5-Mini and Gemini-2.5-Flash.