LLM-Mediated Guidance of MARL Systems

TL;DR

This paper investigates guiding multi-agent reinforcement learning (MARL) with large language model (LLM) interventions in a challenging aerial wildfire suppression task. It introduces two controller types—Rule-Based (RB) and Natural Language (NL)—and a central LLM-Mediator that can override agents' actions to shape learning trajectories, with a focus on early interventions. Experimental results show both RB and NL interventions outperform a no-intervention baseline, with NL prompting stronger improvements in coordination and efficiency, and the approach remaining effective as team size scales. The work demonstrates the viability of real-time, language-enabled guidance to accelerate MARL training and adapt to dynamic, high-dimensional environments, suggesting broad applicability beyond wildfire scenarios.

Abstract

In complex multi-agent environments, achieving efficient learning and desirable behaviours is a significant challenge for Multi-Agent Reinforcement Learning (MARL) systems. This work explores the potential of combining MARL with Large Language Model (LLM)-mediated interventions to guide agents toward more desirable behaviours. Specifically, we investigate how LLMs can be used to interpret and facilitate interventions that shape the learning trajectories of multiple agents. We experimented with two types of interventions, referred to as controllers: a Natural Language (NL) Controller and a Rule-Based (RB) Controller. The NL Controller, which uses an LLM to simulate human-like interventions, showed a stronger impact than the RB Controller. Our findings indicate that agents particularly benefit from early interventions, leading to more efficient training and higher performance. Both intervention types outperform the baseline without interventions, highlighting the potential of LLM-mediated guidance to accelerate training and enhance MARL performance in challenging environments.

Figures (50)

• Figure 1: The Aerial Wildfire Suppression environment includes two types of controllers: Natural Language-based and Rule-Based. Controller interventions are passed to the LLM-Mediator, temporarily providing actions and overwriting the agents' learned policy actions.
• Figure 2: AWS Environment: (1) Water Collection Area, (2) Agent-controlled Wildfire Suppression Aeroplanes, (3) Human Natural Language Controller Input Field, (4) Village. Environment Features: Wind, overcast, temperature and humidity map sample.
• Figure 3: AWS Process Diagram: The default setup consists of three agents controlling individual aeroplanes. Each agent receives both feature vector and visual observations. Agents' actions include steering left, right, or releasing water. Rewards are given for extinguishing burning trees; smaller rewards are given for wetting living trees and picking up water. A negative reward is given for crossing the environment boundary. The LLM-Mediator interprets RB and NL Controller interventions, assigning tasks to any agent for the next 300 steps and overwriting its policy actions.
• Figure 4: Overview of simplified RB and NL Controller intervention prompts sent to the LLM-Mediator, overwriting the agents' learned policy actions.
• Figure 5: Abbreviated Rule-Based Controller intervention prompt template. A complete version can be found in the Appendix \ref{['appendix:RB_prompt_template']}.