Continual Deep Reinforcement Learning for Decentralized Satellite Routing

TL;DR

This work tackles continual decentralized routing in dynamic LEO satellite constellations where per-satellite knowledge is partial and links/buffers fluctuate. It proposes a two-phase framework: offline global training of a Q-network and online on-board continual learning, with two update modes—Model Anticipation for short-term adaptation and Satellite Federated Learning for long-term alignment. The method models routing as a multi-agent reinforcement learning problem using DDQN-style updates and SARS experiences, achieving near-optimal end-to-end latency with limited feedback compared to a centralized shortest-path baseline, and demonstrating robustness under congestion and topology changes. The proposed hierarchical continual-learning scheme offers scalable, energy-efficient onboard deployment and practical applicability to evolving space networks, enabling resilient, adaptive routing in future large-scale satellite constellations.

Abstract

This paper introduces a full solution for decentralized routing in Low Earth Orbit satellite constellations based on continual Deep Reinforcement Learning (DRL). This requires addressing multiple challenges, including the partial knowledge at the satellites and their continuous movement, and the time-varying sources of uncertainty in the system, such as traffic, communication links, or communication buffers. We follow a multi-agent approach, where each satellite acts as an independent decision-making agent, while acquiring a limited knowledge of the environment based on the feedback received from the nearby agents. The solution is divided into two phases. First, an offline learning phase relies on decentralized decisions and a global Deep Neural Network (DNN) trained with global experiences. Then, the online phase with local, on-board, and pre-trained DNNs requires continual learning to evolve with the environment, which can be done in two different ways: (1) Model anticipation, where the predictable conditions of the constellation are exploited by each satellite sharing local model with the next satellite; and (2) Federated Learning (FL), where each agent's model is merged first at the cluster level and then aggregated in a global Parameter Server. The results show that, without high congestion, the proposed Multi-Agent DRL framework achieves the same E2E performance as a shortest-path solution, but the latter assumes intensive communication overhead for real-time network-wise knowledge of the system at a centralized node, whereas ours only requires limited feedback exchange among first neighbour satellites. Importantly, our solution adapts well to congestion conditions and exploits less loaded paths. Moreover, the divergence of models over time is easily tackled by the synergy between anticipation, applied in short-term alignment, and FL, utilized for long-term alignment.

Figures (11)

• Figure 1: Proposed solutions for continual decentralized learning in satellite constellations. (a) Model anticipation: Each agent $i$ transmits its Q-Network, $Q_i(\theta)$, to agent $i-1$ within the orbital plane and moving in the direction opposite to the orbital motion. (b) Satellite Federated Learning (SFL): Each orbital plane is a learning cluster formed by a set of satellite agents. The at a satellite aggregates the cluster models. (c) Detail of the cluster and network aggregation in SFL. The receives each cluster model $Q^o(\theta)$ and aggregate into the global model $Q_g(\theta)$.
• Figure 2: Representation of the four tested constellations and their corresponding established following the Greedy matching over the population maps CIESIN. Each colour is a different orbital plane.
• Figure 3: Proposed learning framework, where: a) presents the environment with a Kepler constellation, including the network topology; b) represents a network-agent interaction needed to build the tuple experience ($\text{SARS}$); c) represents the offline phase; and d) represents the online phase.
• Figure 4: latency and exploration rate versus time during the offline phase with 2 active gateways. Comparison of with the Q-routing algorithm soret2023q and the Shortest path in the four defined constellation architectures at their initial deployment conditions.
• Figure 5: Boxplot of the latency of the four constellation topologies with the Shortest Path policy after one orbital period is completed.