Network Topology Optimization via Deep Reinforcement Learning

TL;DR

A novel deep reinforcement learning algorithm for graph searching, called DRL-GS, for network topology optimization, which can efficiently search over relatively large topology space and output topology with satisfactory performance.

Abstract

Topology impacts important network performance metrics, including link utilization, throughput and latency, and is of central importance to network operators. However, due to the combinatorial nature of network topology, it is extremely difficult to obtain an optimal solution, especially since topology planning in networks also often comes with management-specific constraints. As a result, local optimization with hand-tuned heuristic methods from human experts is often adopted in practice. Yet, heuristic methods cannot cover the global topology design space while taking into account constraints, and cannot guarantee to find good solutions. In this paper, we propose a novel deep reinforcement learning (DRL) algorithm for graph searching, called DRL-GS, for network topology optimization. DRL-GS consists of three novel components, including a verifier to validate the correctness of a generated network topology, a graph neural network (GNN) to efficiently approximate topology rating, and a DRL agent to conduct a topology search. DRL-GS can efficiently search over relatively large topology space and output topology with satisfactory performance. We conduct a case study based on a real-world network scenario, and our experimental results demonstrate the superior performance of DRL-GS in terms of both efficiency and performance.

Figures (11)

• Figure 1: An example of a network topology. There are different node types and the formation of the network is constrained by the management requirements.
• Figure 2: The procedure of DRL-GS. There are three main components, i.e., the representation layer, the DRL agent and the verifier. Here $s$ denotes a state (topology), $a$ denotes an action (link change), $r$ denotes a reward function (score of topology). $T(s, a)$ denotes the transition function. $\pi_\theta$ denotes a parameterized policy. $f(s)$ is the objective value calculated by the verifier. DRL-GS utilizes the representation layer to compress the state and action to $s_0$ and $a_0$ and the GNN classifier $\tilde{f}$ to learn the true objective function $f$.
• Figure 3: Schematic diagram of action compression in five steps.
• Figure 4: Implementation details in GNN for classification.
• Figure 5: Entropy loss (a,b) and value loss (c,d) of RL training in small dataset using A2C (a,c) and PPO (b,d). The red loss curve captures the agents trained in the full space. The blue loss curve captures the agents trained in compressed space (AC means action compression). The green loss curve captures the agents trained in compressed space with GNN classifier.