Multi-source Plume Tracing via Multi-Agent Reinforcement Learning

TL;DR

This paper addresses the challenge of rapidly locating multiple airborne pollution sources under turbulent dispersion. It introduces a MARL framework framed as a Partially Observable Markov Game and proposes ADDRQN, an LSTM-based Q-network that conditions on action histories to handle partial observability in a 3D Gaussian plume environment. The authors demonstrate, through a GPM-based training setup and comparisons to DRQN, DDRQN, and ADRQN, that ADDRQN achieves high success rates, faster convergence, and scalable performance, including locating sources after exploring as little as 1.29% of the environment. The findings confirm the effectiveness of action-history–aware, multi-agent cooperation for efficient plume source localization, with clear benefits as the agent count increases and significant implications for emergency response and environmental monitoring.

Abstract

Industrial catastrophes like the Bhopal disaster (1984) and the Aliso Canyon gas leak (2015) demonstrate the urgent need for rapid and reliable plume tracing algorithms to protect public health and the environment. Traditional methods, such as gradient-based or biologically inspired approaches, often fail in realistic, turbulent conditions. To address these challenges, we present a Multi-Agent Reinforcement Learning (MARL) algorithm designed for localizing multiple airborne pollution sources using a swarm of small uncrewed aerial systems (sUAS). Our method models the problem as a Partially Observable Markov Game (POMG), employing a Long Short-Term Memory (LSTM)-based Action-specific Double Deep Recurrent Q-Network (ADDRQN) that uses full sequences of historical action-observation pairs, effectively approximating latent states. Unlike prior work, we use a general-purpose simulation environment based on the Gaussian Plume Model (GPM), incorporating realistic elements such as a three-dimensional environment, sensor noise, multiple interacting agents, and multiple plume sources. The incorporation of action histories as part of the inputs further enhances the adaptability of our model in complex, partially observable environments. Extensive simulations show that our algorithm significantly outperforms conventional approaches. Specifically, our model allows agents to explore only 1.29\% of the environment to successfully locate pollution sources.

Figures (7)

• Figure 1: Plume dispersion simulations using Eq. \ref{['eq:multiple-point-source']} with two point source emitters, $C_1$ and $C_2$, characterized by distinct wind speeds ($u$), emission rates ($Q$) and effective heights ($H$). For $C_1$, $u = 10$ m/s, $Q=10$ g/s, $H=16$ m, under stability class A conditions; for $C_2$, $u = 8$ m/s, $Q=12$ g/s, $H=16$ m, under stability class B conditions. These meteorological parameters influence the plume behavior and dispersion patterns observed.
• Figure 2: Structure of a Partially Observable Markov Game (POMG). Individual decisions from agents contribute to a joint action vector $\mathbf{a}$ which influences the shared environment. This results in a joint observation vector and an immediate global reward for the agents.
• Figure 3: Recurrent Q-network architecture over two sequential time steps. The network integrates fully connected layers with an LSTM unit to process observations ($\vec{o}_t$) and preceding actions ($a^i_{t-1}$). This allows for estimating Q-values and choosing subsequent actions while maintaining temporal dependencies across episodes. Architecture adapted from zhu2018improving.
• Figure 4:
• Figure 5: Agent trajectories under varying plume dispersion conditions with adjacent color bar indicating corresponding pollution level.