Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FIRE-VLM: A Vision-Language-Driven Reinforcement Learning Framework for UAV Wildfire Tracking in a Physics-Grounded Fire Digital Twin

Chris Webb, Mobin Habibpour, Mayamin Hamid Raha, Ali Reza Tavakkoli, Janice Coen, Fatemeh Afghah

TL;DR

FIRE-VLM tackles autonomous wildfire monitoring by enabling UAVs to operate under severe visual degradation and sparse real-world data. It combines PPO-based reinforcement learning with a vision–language model that semantically interprets wildfire imagery inside a physics-grounded wildfire digital twin, guiding the agent via semantic alignment and directional cues. Key contributions include a GIS-to-simulation pipeline for wildfire twins, a VLM-guided RL agent for firefront tracking, and a wildfire-aware reward that fuses physical objectives with semantic signals. The approach yields up to a sixfold reduction in time-to-detection and improved time-in-FOV across kilometer-scale, physics-grounded fires, illustrating a scalable path toward safer, more effective autonomous wildfire surveillance and response.

Abstract

Wildfire monitoring demands autonomous systems capable of reasoning under extreme visual degradation, rapidly evolving physical dynamics, and scarce real-world training data. Existing UAV navigation approaches rely on simplified simulators and supervised perception pipelines, and lack embodied agents interacting with physically realistic fire environments. We introduce FIRE-VLM, the first end-to-end vision-language model (VLM) guided reinforcement learning (RL) framework trained entirely within a high-fidelity, physics-grounded wildfire digital twin. Built from USGS Digital Elevation Model (DEM) terrain, LANDFIRE fuel inventories, and semi-physical fire-spread solvers, this twin captures terrain-induced runs, wind-driven acceleration, smoke plume occlusion, and dynamic fuel consumption. Within this environment, a PPO agent with dual-view UAV sensing is guided by a CLIP-style VLM. Wildfire-specific semantic alignment scores, derived from a single prompt describing active fire and smoke plumes, are integrated as potential-based reward shaping signals. Our contributions are: (1) a GIS-to-simulation pipeline for constructing wildfire digital twins; (2) a VLM-guided RL agent for UAV firefront tracking; and (3) a wildfire-aware reward design that combines physical terms with VLM semantics. Across five digital-twin evaluation tasks, our VLM-guided policy reduces time-to-detection by up to 6 times, increases time-in-FOV, and is, to our knowledge, the first RL-based UAV wildfire monitoring system demonstrated in kilometer-scale, physics-grounded digital-twin fires.

FIRE-VLM: A Vision-Language-Driven Reinforcement Learning Framework for UAV Wildfire Tracking in a Physics-Grounded Fire Digital Twin

TL;DR

FIRE-VLM tackles autonomous wildfire monitoring by enabling UAVs to operate under severe visual degradation and sparse real-world data. It combines PPO-based reinforcement learning with a vision–language model that semantically interprets wildfire imagery inside a physics-grounded wildfire digital twin, guiding the agent via semantic alignment and directional cues. Key contributions include a GIS-to-simulation pipeline for wildfire twins, a VLM-guided RL agent for firefront tracking, and a wildfire-aware reward that fuses physical objectives with semantic signals. The approach yields up to a sixfold reduction in time-to-detection and improved time-in-FOV across kilometer-scale, physics-grounded fires, illustrating a scalable path toward safer, more effective autonomous wildfire surveillance and response.

Abstract

Wildfire monitoring demands autonomous systems capable of reasoning under extreme visual degradation, rapidly evolving physical dynamics, and scarce real-world training data. Existing UAV navigation approaches rely on simplified simulators and supervised perception pipelines, and lack embodied agents interacting with physically realistic fire environments. We introduce FIRE-VLM, the first end-to-end vision-language model (VLM) guided reinforcement learning (RL) framework trained entirely within a high-fidelity, physics-grounded wildfire digital twin. Built from USGS Digital Elevation Model (DEM) terrain, LANDFIRE fuel inventories, and semi-physical fire-spread solvers, this twin captures terrain-induced runs, wind-driven acceleration, smoke plume occlusion, and dynamic fuel consumption. Within this environment, a PPO agent with dual-view UAV sensing is guided by a CLIP-style VLM. Wildfire-specific semantic alignment scores, derived from a single prompt describing active fire and smoke plumes, are integrated as potential-based reward shaping signals. Our contributions are: (1) a GIS-to-simulation pipeline for constructing wildfire digital twins; (2) a VLM-guided RL agent for UAV firefront tracking; and (3) a wildfire-aware reward design that combines physical terms with VLM semantics. Across five digital-twin evaluation tasks, our VLM-guided policy reduces time-to-detection by up to 6 times, increases time-in-FOV, and is, to our knowledge, the first RL-based UAV wildfire monitoring system demonstrated in kilometer-scale, physics-grounded digital-twin fires.
Paper Structure (30 sections, 10 equations, 10 figures, 9 tables, 1 algorithm)

This paper contains 30 sections, 10 equations, 10 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. UAV Coordination and Decentralized Wildfire Monitoring
  4. RGB/IR Datasets for Supervised Wildfire Detection
  5. Wildfire Digital Twins
  6. RL and Semantic Guidance in Embodied AI
  7. System Overview
  8. Digital Twin Environment
  9. POMDP Formulation and UAV Control
  10. Base Reward Design.
  11. VLM-Guided Reward Shaping
  12. Angled-view potential shaping.
  13. Top-down directional guidance.
  14. Training Pipeline
  15. Experimental Results and Evaluation
  16. ...and 15 more sections

Figures (10)

  • Figure 1: System overview of the proposed VLM-guided UAV wildfire monitoring framework. A physics-grounded wildfire digital twin (terrain, fuel, wind, and fire spread) is simulated in AirSim, producing dual-view RGB observations from top-down and angled cameras. A pretrained VLM computes semantic alignment and directional wildfire likelihood, which shape the PPO policy’s reward and guide the UAV’s 3D motion commands for robust firefront tracking.
  • Figure 2: King Fire side-by-side comparison. Natural-color Landsat 8 imagery of the fire is displayed in the left column as a wider satellite view at the bottom and a close-up at the top nasa_king_fire_image. The equivalent digital twin-based aerial reconstruction produced using CAWFE simulation results is displayed in the right column as (top) close-up and (bottom) satellite view.
  • Figure 3: Top left: top-down UAV camera view with red crosshairs indicating the four directional quadrants (forward, backward, left, right). Top right: corresponding cropped patches extracted from each quadrant, which are passed independently to the VLM to estimate wildfire likelihood in each direction. Bottom left: VLM directional guidance for left & right patches. Bottom right: VLM directional guidance for forward & backward patches.
  • Figure 4: Qualitative episode showing the progression of the VLM-guided UAV policy from exploration to sustained tracking. Each panel shows a composite of the top-down (top) and angled (bottom) RGB views at a representative timestep: (a) initial search with no wildfire in the FOV, (b) first distant smoke cue used by the VLM to steer the agent, (c) locked-on view of the active firefront, and (d) sustained tracking while following the fire perimeter.
  • Figure 5: UAV deployed with wildfire in the initial FOV of its top-down camera (Wildfire in FOV).
  • ...and 5 more figures