Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Prompts to Pavement: LMMs-based Agentic Behavior-Tree Generation Framework for Autonomous Vehicles

Omar Y. Goba, Ahmed Y. Gado, Catherine M. Elias, Ahmed Hussein

TL;DR

This work addresses the rigidity of static behavior trees in autonomous driving by introducing an agentic framework that leverages large language and multimodal models to generate and adapt BTs on demand. A three-agent MAS (Descriptor, Planner, Generator) operates inside a ROS2-CARLA Nav2 stack and is activated when a baseline BT fails, producing executable XML BTs to handle unforeseen obstacles. In CARLA simulation, the framework successfully navigates around a blocked-lane scenario, demonstrating end-to-end adaptability and providing a proof-of-concept for L5-like autonomous behavior. While promising, the approach faces latency, generalization, and verification challenges, motivating future work on local LLM deployment, BT formal verification, and broader real-world validation.

Abstract

Autonomous vehicles (AVs) require adaptive behavior planners to navigate unpredictable, real-world environments safely. Traditional behavior trees (BTs) offer structured decision logic but are inherently static and demand labor-intensive manual tuning, limiting their applicability at SAE Level 5 autonomy. This paper presents an agentic framework that leverages large language models (LLMs) and multi-modal vision models (LVMs) to generate and adapt BTs on the fly. A specialized Descriptor agent applies chain-of-symbols prompting to assess scene criticality, a Planner agent constructs high-level sub-goals via in-context learning, and a Generator agent synthesizes executable BT sub-trees in XML format. Integrated into a CARLA+Nav2 simulation, our system triggers only upon baseline BT failure, demonstrating successful navigation around unexpected obstacles (e.g., street blockage) with no human intervention. Compared to a static BT baseline, this approach is a proof-of-concept that extends to diverse driving scenarios.

From Prompts to Pavement: LMMs-based Agentic Behavior-Tree Generation Framework for Autonomous Vehicles

TL;DR

This work addresses the rigidity of static behavior trees in autonomous driving by introducing an agentic framework that leverages large language and multimodal models to generate and adapt BTs on demand. A three-agent MAS (Descriptor, Planner, Generator) operates inside a ROS2-CARLA Nav2 stack and is activated when a baseline BT fails, producing executable XML BTs to handle unforeseen obstacles. In CARLA simulation, the framework successfully navigates around a blocked-lane scenario, demonstrating end-to-end adaptability and providing a proof-of-concept for L5-like autonomous behavior. While promising, the approach faces latency, generalization, and verification challenges, motivating future work on local LLM deployment, BT formal verification, and broader real-world validation.

Abstract

Autonomous vehicles (AVs) require adaptive behavior planners to navigate unpredictable, real-world environments safely. Traditional behavior trees (BTs) offer structured decision logic but are inherently static and demand labor-intensive manual tuning, limiting their applicability at SAE Level 5 autonomy. This paper presents an agentic framework that leverages large language models (LLMs) and multi-modal vision models (LVMs) to generate and adapt BTs on the fly. A specialized Descriptor agent applies chain-of-symbols prompting to assess scene criticality, a Planner agent constructs high-level sub-goals via in-context learning, and a Generator agent synthesizes executable BT sub-trees in XML format. Integrated into a CARLA+Nav2 simulation, our system triggers only upon baseline BT failure, demonstrating successful navigation around unexpected obstacles (e.g., street blockage) with no human intervention. Compared to a static BT baseline, this approach is a proof-of-concept that extends to diverse driving scenarios.
Paper Structure (18 sections, 6 equations, 3 figures, 1 table)

This paper contains 18 sections, 6 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. State-of-the Art
  3. Methodology
  4. System Overview
  5. Simulation Environment Preparation
  6. SLAM Layer
  7. Perception, Navigation and Control Layers
  8. Overall System Flow
  9. Agentic Behavior-Tree Generation Framework
  10. The Descriptor Agent
  11. The Planner Agent
  12. The Generator Agent
  13. Agents' Configuration
  14. Results and Discussion
  15. Evaluation of scene interpretation (Descriptor)
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The Overall ROS-Based Autonomous Driving Stack with the Agentic Behavior-Tree Generation Framework
  • Figure 2: Vehicle's Initial Position and Initial Scene Description with Obstructions and Goal Pose
  • Figure 3: The Behavior Trees (a) Baseline, (b) Prompt to Pavement Framework Generated