HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments

TL;DR

HAZARD tackles embodied decision-making in dynamically changing environments by introducing a ThreeDWorld-based benchmark with three disaster types (fire, flood, wind) and an LLM-enabled decision API. It designs a hierarchical LLM-based planning pipeline that converts perception and memory into text prompts to select high-level actions, while employing A* navigation to compress trajectories and reduce query frequency. The benchmark includes procedurally generated scenes and evaluative metrics (Value, Step, Damage) across multiple baselines (Random, Rule-based, MCTS, PPO-based RL) and LLM backbones (e.g., GPT-4), enabling cross-method comparisons. Findings indicate GPT-4 yields strong zero-shot performance but struggles with complex environmental dynamics, highlighting ongoing challenges in perception and reasoning under change and motivating further enhancements in action spaces and planning under dynamic hazards.

Abstract

Recent advances in high-fidelity virtual environments serve as one of the major driving forces for building intelligent embodied agents to perceive, reason and interact with the physical world. Typically, these environments remain unchanged unless agents interact with them. However, in real-world scenarios, agents might also face dynamically changing environments characterized by unexpected events and need to rapidly take action accordingly. To remedy this gap, we propose a new simulated embodied benchmark, called HAZARD, specifically designed to assess the decision-making abilities of embodied agents in dynamic situations. HAZARD consists of three unexpected disaster scenarios, including fire, flood, and wind, and specifically supports the utilization of large language models (LLMs) to assist common sense reasoning and decision-making. This benchmark enables us to evaluate autonomous agents' decision-making capabilities across various pipelines, including reinforcement learning (RL), rule-based, and search-based methods in dynamically changing environments. As a first step toward addressing this challenge using large language models, we further develop an LLM-based agent and perform an in-depth analysis of its promise and challenge of solving these challenging tasks. HAZARD is available at https://vis-www.cs.umass.edu/hazard/.

Figures (7)

• Figure 1: Illustration of HAZARD Challenge. The HAZARD challenge consists of three dynamically changing scenarios: fire , flood , and wind . In the fire scenario, flames continuously spread and burn objects. In the flood scenario, water spreads and rises, washing away objects and causing damage to non-waterproof objects. The wind scenario poses the challenge of objects being blown away, making them hard to reach. These scenarios present embodied agents with complex perception, reasoning, and planning challenges.
• Figure 2: Detailed visual effect of fire and flood . We have developed near-realistic visual effects for the fire and flood scenarios, which are controlled by our simulation system.
• Figure 3: Benchmark details. In HAZARD challenge, an embodied agent needs to rescue a given set of objects from disasters. The agent observations include RGB-D signals, temperature or water level signals, target object information, and segmentation masks. To address the challenge in perception, we also provide a perceptional version of HAZARD which excludes segmentation mask from observations. The action space consists of four high-level actions: Pick Up, Explore, Drop, and Walk To, each representing a compression of multiple low-level actions. The final performance of agents is measured by 'value', 'step', and 'damage'.
• Figure 4: Framework of the proposed LLM-based pipeline. The LLM takes in diverse input information from the environment and engages in comprehensive decision-making. At the high level, the LLM selects actions such as "walk to", "pick up", "drop", or "explore". These chosen actions are then executed through a series of low-level actions. This hierarchical approach enables the LLM to effectively process and utilize the various types of input information available.
• Figure 5: A qualitative result of the LLM pipeline. The GPT-4 model takes simple attributes, such as distance, temperature, and object value into consideration to enhance its decision-making abilities.