Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Deep Reinforcement Learning for Distributed Satellite Routing

Federico Lozano-Cuadra, Beatriz Soret

TL;DR

The paper addresses end-to-end routing in Low Earth Orbit Satellite Constellations where moving topology, propagation delays, and dynamic traffic complicate routing. It proposes a Multi-Agent Deep Reinforcement Learning (MA-DRL) framework in which each satellite is an independent agent with partial environment knowledge, coordinated through local feedback, and trained via a two-phase process: offline exploration with a global DNN ($DNN_g$) to learn optimal routes across positions and congestion levels, and online exploitation with onboard copies ($DNN_i$) deployed on each satellite. Routing is formulated as a partially observable Markov decision process ($POMDP$) and uses the SARS experience tuple to collect transitions for training. Results show that offline learning quickly yields effective routing policies and that online, distributed execution with onboard DNNs enables scalable, low-latency routing with minimal information exchange, including the ability to switch to alternative paths under congestion.

Abstract

This paper introduces a Multi-Agent Deep Reinforcement Learning (MA-DRL) approach for routing in Low Earth Orbit Satellite Constellations (LSatCs). Each satellite is an independent decision-making agent with a partial knowledge of the environment, and supported by feedback received from the nearby agents. Building on our previous work that introduced a Q-routing solution, the contribution of this paper is to extend it to a deep learning framework able to quickly adapt to the network and traffic changes, and based on two phases: (1) An offline exploration learning phase that relies on a global Deep Neural Network (DNN) to learn the optimal paths at each possible position and congestion level; (2) An online exploitation phase with local, on-board, pre-trained DNNs. Results show that MA-DRL efficiently learns optimal routes offline that are then loaded for an efficient distributed routing online.

Multi-Agent Deep Reinforcement Learning for Distributed Satellite Routing

TL;DR

The paper addresses end-to-end routing in Low Earth Orbit Satellite Constellations where moving topology, propagation delays, and dynamic traffic complicate routing. It proposes a Multi-Agent Deep Reinforcement Learning (MA-DRL) framework in which each satellite is an independent agent with partial environment knowledge, coordinated through local feedback, and trained via a two-phase process: offline exploration with a global DNN () to learn optimal routes across positions and congestion levels, and online exploitation with onboard copies () deployed on each satellite. Routing is formulated as a partially observable Markov decision process () and uses the SARS experience tuple to collect transitions for training. Results show that offline learning quickly yields effective routing policies and that online, distributed execution with onboard DNNs enables scalable, low-latency routing with minimal information exchange, including the ability to switch to alternative paths under congestion.

Abstract

This paper introduces a Multi-Agent Deep Reinforcement Learning (MA-DRL) approach for routing in Low Earth Orbit Satellite Constellations (LSatCs). Each satellite is an independent decision-making agent with a partial knowledge of the environment, and supported by feedback received from the nearby agents. Building on our previous work that introduced a Q-routing solution, the contribution of this paper is to extend it to a deep learning framework able to quickly adapt to the network and traffic changes, and based on two phases: (1) An offline exploration learning phase that relies on a global Deep Neural Network (DNN) to learn the optimal paths at each possible position and congestion level; (2) An online exploitation phase with local, on-board, pre-trained DNNs. Results show that MA-DRL efficiently learns optimal routes offline that are then loaded for an efficient distributed routing online.
Paper Structure (5 sections, 2 figures)

This paper contains 5 sections, 2 figures.

Table of Contents

  1. Introduction
  2. System Model
  3. Learning framework
  4. Preliminary results
  5. Conclusions

Figures (2)

  • Figure 1: System model: 1) network graph; 2) multi-agent interaction needed to build the tuple experience ($\text{SARS}$); 3) and 4) are representations of the offline exploration and online exploitation phases, respectively.
  • Figure 2: E2E latency versus packet creation time. Comparison of our MA-DRL with the Q-Routing algorithm soret2023q and the slant range shortest path benchmark.