Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAPTOR-AI for Disaster OODA Loop: Hierarchical Multimodal RAG with Experience-Driven Agentic Decision-Making

Takato Yasuno

TL;DR

This work presents RAPTOR-AI, an agentic multimodal Retrieval-Augmented Generation framework tailored for multi-stage disaster response, integrating a hierarchical multimodal knowledge tree with an entropy-aware retrieval controller and LoRA-based experiential learning. By combining RAPTOR's recursive abstraction, ColVBERT-driven retrieval, and BLIP-based vision-language grounding, the system achieves improved retrieval quality, situational grounding, and task decomposition across initial rescue, mid-term recovery, and long-term reconstruction. Experimental results on a tsunami-focused corpus show substantial gains in top-k retrieval accuracy, grounding, and operator usability, along with ablation evidence of each component's value. The approach emphasizes open-source reproducibility, real-time adaptability, and region-specific customization, offering a scalable foundation for global disaster intelligence and practical HADR deployment.

Abstract

Effective humanitarian assistance and disaster relief (HADR) requires rapid situational understanding, reliable decision support, and the ability to generalize across diverse and previously unseen disaster contexts. This work introduces an agentic Retrieval-Augmented Generation (RAG) framework designed to support the three canonical phases of disaster response: initial rescue, mid-term recovery, and long-term reconstruction. To achieve robust multimodal grounding, we construct a hierarchical knowledge base that integrates textual disaster manuals, historical lessons (e.g., the 2011 Tohoku earthquake), and both aerial and ground-level imagery. Our system builds on the open-source multimodal implementation, which processes 46 tsunami-related PDFs (2,378 pages) using BLIP-based image captioning, ColVBERT embeddings, and long-context summarization to generate an efficient, structured multimodal retrieval tree optimized for disaster knowledge preservation. An agentic controller dynamically selects retrieval strategies (e.g., RAPTOR, ColBERT) through entropy-aware scene abstraction, enabling adaptive reasoning across heterogeneous inputs. Additionally, a lightweight LoRA-based post-training method injects experiential knowledge from past disasters, enhancing the models' capacity to support both expert and non-expert responders. Experiments on real disaster datasets demonstrate improved situational grounding, enhanced task decomposition accuracy, and superior usability for emergency operations. Incorporating recent advances in long-context RAG systems, agentic information retrieval, and contemporary emergency response AI, our system achieves substantial gains through adaptive retrieval-augmented generation with self-reasoning and multimodal chain-of-thought capabilities.

RAPTOR-AI for Disaster OODA Loop: Hierarchical Multimodal RAG with Experience-Driven Agentic Decision-Making

TL;DR

This work presents RAPTOR-AI, an agentic multimodal Retrieval-Augmented Generation framework tailored for multi-stage disaster response, integrating a hierarchical multimodal knowledge tree with an entropy-aware retrieval controller and LoRA-based experiential learning. By combining RAPTOR's recursive abstraction, ColVBERT-driven retrieval, and BLIP-based vision-language grounding, the system achieves improved retrieval quality, situational grounding, and task decomposition across initial rescue, mid-term recovery, and long-term reconstruction. Experimental results on a tsunami-focused corpus show substantial gains in top-k retrieval accuracy, grounding, and operator usability, along with ablation evidence of each component's value. The approach emphasizes open-source reproducibility, real-time adaptability, and region-specific customization, offering a scalable foundation for global disaster intelligence and practical HADR deployment.

Abstract

Effective humanitarian assistance and disaster relief (HADR) requires rapid situational understanding, reliable decision support, and the ability to generalize across diverse and previously unseen disaster contexts. This work introduces an agentic Retrieval-Augmented Generation (RAG) framework designed to support the three canonical phases of disaster response: initial rescue, mid-term recovery, and long-term reconstruction. To achieve robust multimodal grounding, we construct a hierarchical knowledge base that integrates textual disaster manuals, historical lessons (e.g., the 2011 Tohoku earthquake), and both aerial and ground-level imagery. Our system builds on the open-source multimodal implementation, which processes 46 tsunami-related PDFs (2,378 pages) using BLIP-based image captioning, ColVBERT embeddings, and long-context summarization to generate an efficient, structured multimodal retrieval tree optimized for disaster knowledge preservation. An agentic controller dynamically selects retrieval strategies (e.g., RAPTOR, ColBERT) through entropy-aware scene abstraction, enabling adaptive reasoning across heterogeneous inputs. Additionally, a lightweight LoRA-based post-training method injects experiential knowledge from past disasters, enhancing the models' capacity to support both expert and non-expert responders. Experiments on real disaster datasets demonstrate improved situational grounding, enhanced task decomposition accuracy, and superior usability for emergency operations. Incorporating recent advances in long-context RAG systems, agentic information retrieval, and contemporary emergency response AI, our system achieves substantial gains through adaptive retrieval-augmented generation with self-reasoning and multimodal chain-of-thought capabilities.
Paper Structure (60 sections, 6 equations, 9 figures, 3 tables)

This paper contains 60 sections, 6 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Overview of RAPTOR-AI for Disaster OODA Loop
  3. Related Work
  4. Retrieval-Augmented Generation Systems
  5. Dense Retrieval and Hierarchical Methods
  6. Multimodal Vision-Language Models
  7. AI Applications in Disaster Response
  8. Agentic AI Systems and Tool Integration
  9. Advances in Long-Context RAG and Adaptive Systems
  10. Contemporary Emergency Response AI Systems
  11. Positioning of Our Work
  12. System Architecture
  13. Data Processing Pipeline
  14. Hierarchical Multimodal Retrieval Tree
  15. Multimodal Embedding Construction
  16. ...and 45 more sections

Figures (9)

  • Figure 1: RAPTOR-AI Framework for Disaster Response OODA Loop. Comprehensive overview of the agentic RAG system supporting the four phases of emergency decision-making: Observe (multimodal data ingestion), Orient (hierarchical knowledge processing), Decide (agentic strategy selection), and Act (contextual response generation). The framework enables rapid transition from initial response to recovery stages through adaptive knowledge retrieval and experiential learning integration.
  • Figure 2: System Architecture for Agentic RAG-based Disaster Response. Complete data flow from document ingestion through multimodal processing to context-aware response generation. The framework processes 46 tsunami-related PDFs (2,378 pages) through six interconnected layers with BLIP-2 image captioning, ColVBERT embeddings, and GPT-based summarization.
  • Figure 3: 1,000 chunks
  • Figure 4: 2,000 chunks
  • Figure 5: 3,000 chunks
  • ...and 4 more figures