A RAG-Based Multi-Agent LLM System for Natural Hazard Resilience and Adaptation

TL;DR

This work tackles the challenge of context-specific, location-aware wildfire risk decision-support by introducing WildfireGPT, a retrieval-augmented, multi-agent LLM system. The architecture combines a user_profile_agent, planning_agent, and analyst_agent under a task_orchestrator to personalize analysis and recommendations through data fusion of hazard projections, observations, demographics, and literature, with interactive geospatial visualizations. A three-stage evaluation framework—data/literature retrieval comparison, personalization ablation, and domain-expert plus LLM-as-a-judge assessments—demonstrates that WildfireGPT outperforms baseline tools in data provision, location specificity, and data accuracy, while delivering high contextual relevance. The study also explores LLM-based automated evaluation (LLM-as-a-Judge) and scalable evaluation considerations, offering insights into deployment-time quality assurance and future directions for adaptive, domain-focused AI-assisted hazard decision-making. Overall, the system represents a pragmatic, scalable approach to integrating localized data and expert knowledge to support resilient adaptation to natural hazards across diverse stakeholder groups.

Abstract

Large language models (LLMs) are a transformational capability at the frontier of artificial intelligence and machine learning that can support decision-makers in addressing pressing societal challenges such as extreme natural hazard events. As generalized models, LLMs often struggle to provide context-specific information, particularly in areas requiring specialized knowledge. In this work we propose a retrieval-augmented generation (RAG)-based multi-agent LLM system to support analysis and decision-making in the context of natural hazards and extreme weather events. As a proof of concept, we present WildfireGPT, a specialized system focused on wildfire hazards. The architecture employs a user-centered, multi-agent design to deliver tailored risk insights across diverse stakeholder groups. By integrating natural hazard and extreme weather projection data, observational datasets, and scientific literature through an RAG framework, the system ensures both the accuracy and contextual relevance of the information it provides. Evaluation across ten expert-led case studies demonstrates that WildfireGPT significantly outperforms existing LLM-based solutions for decision support.

Figures (8)

• Figure 1: Overview of WildfireGPT architecture comprising an LLM agent integrated with the multi-agent RAG framework. The WildfireGPT prototype focuses on enhancing consulting interactions using LLM agents stepping through a multistage approach. Its user profile agent engages the user with a tailored questionnaire to create a detailed profile; the planning agent formulates a customized action plan addressing the user's queries and concerns, ensuring alignment with their evolving needs; and the analyst agent aids in accessing and interpreting relevant data and literature and providing recommendations.
• Figure 2: Interactive visualizations in the WildfireGPT user experience. This example is taken from the case study themed "Current Wildfire Risk Analysis." Users can select the season and time period to view the corresponding Fire Weather Index (FWI) map (left of first row), which displays risk levels using a color scale. Location-specific FWI values are accessible by hovering over the map (center of first row). By selecting different seasons and time period, the users can comprehend the changing landscape of the FWI (center of first row to center of second row). The wildfire incident map (right of the second row) shows the spatial distribution of recent fires, while the line graph (bottom left) presents the temporal trend of incidents. Socioeconomic data is visualized through census block group overlays (bottom right), providing insights into poverty rates and housing units in each area.
• Figure 3: Ecosystem Fire Management: WildfireGPT demonstrates its ability to integrate data analysis and domain knowledge to provide actionable recommendations for ecosystem fire management. By analyzing wildfire incident data, retrieving relevant literature, and examining seasonal FWI trends, WildfireGPT suggests optimizing the timing of controlled burns to minimize risks while maintaining oak ecosystem health. The domain expert's positive feedback highlights WildfireGPT's nuanced approach and its potential to support informed decision-making in wildfire management.
• Figure 4: Wildland Urban Interface Impact: WildfireGPT demonstrates its effectiveness in addressing the complex challenges posed by the wildland-urban interface. Wildfire's recommendations regarding water quality protection align well with the actual challenges faced by domain experts in the field. WildfireGPT's ability to identify relevant case studies from other areas with similar shifts in wildfire risk and to highlight pertinent mitigation strategies showcases its potential to support informed decision-making in wildfire risk management and urban planning.
• Figure 5: Overview of the WildfireGPT user experience. The screenshots are taken from one of the case studies themed Comprehensive Wildfire Impact. The user profile agent (top left) engages the user in a conversation to understand the user's background and concerns. The planning agent (top middle) generates a tailored analysis plan based on the user's profile. The analyst agent then executes the plan, analyzing Fire Weather Index data (top right and bottom left), conducting a literature review (bottom middle), and generating personalized recommendations (bottom right) to address the user's wildfire risk concerns.