Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems

Tianyang Duan, Zongyuan Zhang, Zheng Lin, Songxiao Guo, Xiuxian Guan, Guangyu Wu, Zihan Fang, Haotian Meng, Xia Du, Ji-Zhe Zhou, Heming Cui, Jun Luo, Yue Gao

TL;DR

This work tackles non-stationarity in multi-agent reinforcement learning for edge-enabled mobile systems by introducing RELED, a scalable framework that fuses large language model (LLM) driven expert demonstrations with autonomous agent exploration. It rests on a Stationarity-Aware Demonstration (SED) module that refines LLM prompts using theoretical non-stationarity bounds and a Hybrid Expert-Agent Policy Optimization (HPO) module that adaptively blends expert and agent trajectories via a dynamic DTW-based weight. The approach is formalized on a Dec-POMDP, with a key bound linking external reward volatility and intra-subset policy divergence, guiding demonstration refinement and policy updates. Empirical results on OpenStreetMap-derived traffic networks in SUMO show RELED achieving superior sample and time efficiency, scalability to more agents, and robust generalization compared to state-of-the-art MARL baselines, highlighting its practical potential for real-world decentralized systems such as urban traffic management.

Abstract

Multi-agent reinforcement learning (MARL) has been increasingly adopted in many real-world applications. While MARL enables decentralized deployment on resource-constrained edge devices, it suffers from severe non-stationarity due to the synchronous updates of agent policies. This non stationarity results in unstable training and poor policy con vergence, especially as the number of agents increases. In this paper, we propose RELED, a scalable MARL framework that integrates large language model (LLM)-driven expert demonstrations with autonomous agent exploration. RELED incorporates a Stationarity-Aware Expert Demonstration module, which leverages theoretical non-stationarity bounds to enhance the quality of LLM-generated expert trajectories, thus providing high reward and training-stable samples for each agent. Moreover, a Hybrid Expert-Agent Policy Optimization module adaptively balances each agent's learning from both expert-generated and agent-generated trajectories, accelerating policy convergence and improving generalization. Extensive experiments with real city networks based on OpenStreetMap demonstrate that RELED achieves superior performance compared to state-of-the-art MARL methods.

LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems

TL;DR

This work tackles non-stationarity in multi-agent reinforcement learning for edge-enabled mobile systems by introducing RELED, a scalable framework that fuses large language model (LLM) driven expert demonstrations with autonomous agent exploration. It rests on a Stationarity-Aware Demonstration (SED) module that refines LLM prompts using theoretical non-stationarity bounds and a Hybrid Expert-Agent Policy Optimization (HPO) module that adaptively blends expert and agent trajectories via a dynamic DTW-based weight. The approach is formalized on a Dec-POMDP, with a key bound linking external reward volatility and intra-subset policy divergence, guiding demonstration refinement and policy updates. Empirical results on OpenStreetMap-derived traffic networks in SUMO show RELED achieving superior sample and time efficiency, scalability to more agents, and robust generalization compared to state-of-the-art MARL baselines, highlighting its practical potential for real-world decentralized systems such as urban traffic management.

Abstract

Multi-agent reinforcement learning (MARL) has been increasingly adopted in many real-world applications. While MARL enables decentralized deployment on resource-constrained edge devices, it suffers from severe non-stationarity due to the synchronous updates of agent policies. This non stationarity results in unstable training and poor policy con vergence, especially as the number of agents increases. In this paper, we propose RELED, a scalable MARL framework that integrates large language model (LLM)-driven expert demonstrations with autonomous agent exploration. RELED incorporates a Stationarity-Aware Expert Demonstration module, which leverages theoretical non-stationarity bounds to enhance the quality of LLM-generated expert trajectories, thus providing high reward and training-stable samples for each agent. Moreover, a Hybrid Expert-Agent Policy Optimization module adaptively balances each agent's learning from both expert-generated and agent-generated trajectories, accelerating policy convergence and improving generalization. Extensive experiments with real city networks based on OpenStreetMap demonstrate that RELED achieves superior performance compared to state-of-the-art MARL methods.

Paper Structure

This paper contains 19 sections, 2 theorems, 35 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. RELED Framework
  4. System Model
  5. System Overview
  6. Theoretical Analysis of Non-Stationarity in MARL System
  7. Stationarity-Aware Expert Demonstration Module
  8. Hybrid Expert-Agent Policy Optimization Module
  9. Implementation
  10. Environment Setups
  11. Baselines
  12. RELED Prompt Design
  13. Evaluation
  14. Comparative Results
  15. Sample Analysis of HPO Module
  16. ...and 4 more sections

Key Result

Theorem 1

(Performance Bound on Group-Level Non-Stationarity) Consider two arbitrary sets of joint policies for an agent subset $\mathcal{A}_j$: where $\mathbf{o}^j_t=\left ( o^k_t \right )_{k\in \mathcal{A}_{j}}$ denotes the joint observation of the agent subset $\mathcal{A}_j$. Let $\boldsymbol{\pi}^{{\mathcal{A}_{-j}}} = \left\{\, \pi^i \mid i \in \mathcal{A} \setminus {\mathcal{A}_j} \, \right\}$ denot

Figures (9)

  • Figure 1: An overview of RELED framework.
  • Figure 2: The hybrid policy optimization mechanism adaptively combines expert demonstration guidance (red solid line) and autonomous exploration (black dashed line), facilitating the gradual convergence of agent's policy towards the optimal solution.
  • Figure 3: Experimental scenarios.
  • Figure 4: Sample efficiency across cities and traffic regimes. (a–d) Average episode reward vs. interaction steps under Moderate and Congested settings. (e–h) Average travel time (measured in SUMO simulator time step) vs. steps. Shaded regions denote SD over 3 independent runs.
  • Figure 5: Time efficiency in wall-clock training. Average episode reward over elapsed time for Orlando and Hong Kong (Moderate and Congested). Shaded regions indicate SD across 3 independent runs.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Theorem 1
  • proof
  • Corollary 1
  • proof