Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Signal attenuation enables scalable decentralized multi-agent reinforcement learning over networks

Wesley A Suttle, Vipul K Sharma, Brian M Sadler

TL;DR

This work tackles the scalability barrier of multi-agent reinforcement learning by exploiting signal attenuation (path loss) to enable decentralized policies in networks. It develops two constrained CMAMDP formulations for radar-based power allocation aimed at LPI target detection, and proves that local neighborhood information suffices to approximate global value and gradient expressions with provable error bounds. The authors then formulate and validate decentralized saddle-point policy gradient algorithms that operate using local information within a κ-hop neighborhood, providing explicit gradient approximations and Lagrangian updates for both the sum-SINR maximization and power-minimization-with-SINR-threshold problems. The approach offers a principled blueprint for applying scalable decentralized MARL to wireless communications and radar network problems, with potential extensions to broader networked control tasks where inter-agent influence decays with distance.

Abstract

Multi-agent reinforcement learning (MARL) methods typically require that agents enjoy global state observability, preventing development of decentralized algorithms and limiting scalability. Recent work has shown that, under assumptions on decaying inter-agent influence, global observability can be replaced by local neighborhood observability at each agent, enabling decentralization and scalability. Real-world applications enjoying such decay properties remain underexplored, however, despite the fact that signal power decay, or signal attenuation, due to path loss is an intrinsic feature of many problems in wireless communications and radar networks. In this paper, we show that signal attenuation enables decentralization in MARL by considering the illustrative special case of performing power allocation for target detection in a radar network. To achieve this, we propose two new constrained multi-agent Markov decision process formulations of this power allocation problem, derive local neighborhood approximations for global value function and policy gradient estimates and establish corresponding error bounds, and develop decentralized saddle point policy gradient algorithms for solving the proposed problems. Our approach, though oriented towards the specific radar network problem we consider, provides a useful model for extensions to additional problems in wireless communications and radar networks.

Signal attenuation enables scalable decentralized multi-agent reinforcement learning over networks

TL;DR

This work tackles the scalability barrier of multi-agent reinforcement learning by exploiting signal attenuation (path loss) to enable decentralized policies in networks. It develops two constrained CMAMDP formulations for radar-based power allocation aimed at LPI target detection, and proves that local neighborhood information suffices to approximate global value and gradient expressions with provable error bounds. The authors then formulate and validate decentralized saddle-point policy gradient algorithms that operate using local information within a κ-hop neighborhood, providing explicit gradient approximations and Lagrangian updates for both the sum-SINR maximization and power-minimization-with-SINR-threshold problems. The approach offers a principled blueprint for applying scalable decentralized MARL to wireless communications and radar network problems, with potential extensions to broader networked control tasks where inter-agent influence decays with distance.

Abstract

Multi-agent reinforcement learning (MARL) methods typically require that agents enjoy global state observability, preventing development of decentralized algorithms and limiting scalability. Recent work has shown that, under assumptions on decaying inter-agent influence, global observability can be replaced by local neighborhood observability at each agent, enabling decentralization and scalability. Real-world applications enjoying such decay properties remain underexplored, however, despite the fact that signal power decay, or signal attenuation, due to path loss is an intrinsic feature of many problems in wireless communications and radar networks. In this paper, we show that signal attenuation enables decentralization in MARL by considering the illustrative special case of performing power allocation for target detection in a radar network. To achieve this, we propose two new constrained multi-agent Markov decision process formulations of this power allocation problem, derive local neighborhood approximations for global value function and policy gradient estimates and establish corresponding error bounds, and develop decentralized saddle point policy gradient algorithms for solving the proposed problems. Our approach, though oriented towards the specific radar network problem we consider, provides a useful model for extensions to additional problems in wireless communications and radar networks.
Paper Structure (13 sections, 2 theorems, 46 equations, 1 figure, 3 algorithms)

This paper contains 13 sections, 2 theorems, 46 equations, 1 figure, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Radar network model
  4. Constrained multi-agent MDP formulations
  5. Sum-of-SINRs Maximization
  6. Power Minimization with SINR Threshold
  7. Signal attenuation enables decentralization
  8. MARL Global Observability Issue
  9. Signal Attenuation and Decentralization
  10. Algorithms
  11. Sum-of-SINRs Maximization
  12. Power Minimization with SINR Threshold
  13. Conclusion

Key Result

Theorem 1

Let Assumptions assum:network_coverage, assum:pairwise_cost_independence, assum:regularity_conditions, and assum:uniform_ergodicity hold. For any $\theta \in \Theta$, $i \in \mathcal{N}$, $s^{\mathcal{N}_{\kappa}(i)} \in \mathcal{S}^{\mathcal{N}_{\kappa}(i)}$, $a^{\mathcal{N}_{\kappa}(i)} \in \mathc for all $s^{\mathcal{N}^{-1}_{\kappa}(i)}, \bar{s}^{\mathcal{N}^{-1}_{\kappa}(i)} \in \mathcal{S}^{

Figures (1)

  • Figure 1: Illustrations of radar network system model of Section \ref{['subsec:radar_network_model']} (left) and communication neighborhoods of Section \ref{['subsec:cmamdp']} (right).

Theorems & Definitions (6)

  • Remark 1
  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Remark 2