An open source Multi-Agent Deep Reinforcement Learning Routing Simulator for satellite networks

TL;DR

This work tackles robust packet routing in Low Earth Orbit satellite constellations under movement and traffic uncertainties. It introduces an open-source Python-based simulator with a discrete-event engine that supports Dijkstra-based routing, Q-Routing, and MA-DRL on a time-varying graph $\mathcal{G}_t(\mathcal{N}, \mathcal{E})$. The platform offers configurable constellations (including Walker delta/star designs and real deployments like Kepler, Iridium Next, OneWeb, and Starlink), realistic data-rate models, and visualization/post-processing tools. Experiments show RL-based routing reduces end-to-end latency relative to traditional shortest-path policies, demonstrating practical impact for future satellite networks.

Abstract

This paper introduces an open source simulator for packet routing in Low Earth Orbit Satellite Constellations (LSatCs) considering the dynamic system uncertainties. The simulator, implemented in Python, supports traditional Dijkstra's based routing as well as more advanced learning solutions, specifically Q-Routing and Multi-Agent Deep Reinforcement Learning (MA-DRL) from our previous work. It uses an event-based approach with the SimPy module to accurately simulate packet creation, routing and queuing, providing real-time tracking of queues and latency. The simulator is highly configurable, allowing adjustments in routing policies, traffic, ground and space layer topologies, communication parameters, and learning hyperparameters. Key features include the ability to visualize system motion and track packet paths. Results highlight significant improvements in end-to-end (E2E) latency using Reinforcement Learning (RL)-based routing policies compared to traditional methods. The source code, the documentation and a Jupyter notebook with post-processing results and analysis are available on GitHub.

Figures (7)

• Figure 1: Kepler constellation deployed and their corresponding established following the Greedy matching with 18 active gateways over the population maps CIESIN, where the green tone depends on the population density. Each satellite's colour is a different orbital plane.
• Figure 2: Input-Output Routing Simulator workflow.
• Figure 3: 's exploitation phase congestion test for all routes output with 8 active gateways.
• Figure 4: Rewards over time of the offline phase of with 8 active gateways. The highest rewards are given after a packet has been delivered to the receiving gateway.
• Figure 5: latency vs time vs $\epsilon$ connecting one gateway in Malaga, Spain and another one in Los Angeles, USA, through the Starlink constellation during the offline phase of the -based methods. It can be appreciated how both methods learn to find the optimal path in less than 1 second.