Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse Mean Field Load Balancing in Large Localized Queueing Systems

Anam Tahir, Kai Cui, Heinz Koeppl

TL;DR

This work uses recent advances in sparse mean field theory to learn a near-optimal load balancing policy in sparsely connected queueing networks in a tractable manner, and obtains a general load balancing framework for a large class of sparse bounded-degree wireless topologies.

Abstract

Scalable load balancing algorithms are of great interest in cloud networks and data centers, necessitating the use of tractable techniques to compute optimal load balancing policies for good performance. However, most existing scalable techniques, especially asymptotically scaling methods based on mean field theory, have not been able to model large queueing networks with strong locality. Meanwhile, general multi-agent reinforcement learning techniques can be hard to scale and usually lack a theoretical foundation. In this work, we address this challenge by leveraging recent advances in sparse mean field theory to learn a near-optimal load balancing policy in sparsely connected queueing networks in a tractable manner, which may be preferable to global approaches in terms of wireless communication overhead. Importantly, we obtain a general load balancing framework for a large class of sparse bounded-degree wireless topologies. By formulating a novel mean field control problem in the context of graphs with bounded degree, we reduce the otherwise difficult multi-agent problem to a single-agent problem. Theoretically, the approach is justified by approximation guarantees. Empirically, the proposed methodology performs well on several realistic and scalable wireless network topologies as compared to a number of well-known load balancing heuristics and existing scalable multi-agent reinforcement learning methods.

Sparse Mean Field Load Balancing in Large Localized Queueing Systems

TL;DR

This work uses recent advances in sparse mean field theory to learn a near-optimal load balancing policy in sparsely connected queueing networks in a tractable manner, and obtains a general load balancing framework for a large class of sparse bounded-degree wireless topologies.

Abstract

Scalable load balancing algorithms are of great interest in cloud networks and data centers, necessitating the use of tractable techniques to compute optimal load balancing policies for good performance. However, most existing scalable techniques, especially asymptotically scaling methods based on mean field theory, have not been able to model large queueing networks with strong locality. Meanwhile, general multi-agent reinforcement learning techniques can be hard to scale and usually lack a theoretical foundation. In this work, we address this challenge by leveraging recent advances in sparse mean field theory to learn a near-optimal load balancing policy in sparsely connected queueing networks in a tractable manner, which may be preferable to global approaches in terms of wireless communication overhead. Importantly, we obtain a general load balancing framework for a large class of sparse bounded-degree wireless topologies. By formulating a novel mean field control problem in the context of graphs with bounded degree, we reduce the otherwise difficult multi-agent problem to a single-agent problem. Theoretically, the approach is justified by approximation guarantees. Empirically, the proposed methodology performs well on several realistic and scalable wireless network topologies as compared to a number of well-known load balancing heuristics and existing scalable multi-agent reinforcement learning methods.
Paper Structure (33 sections, 2 theorems, 4 equations, 10 figures, 2 tables, 1 algorithm)

This paper contains 33 sections, 2 theorems, 4 equations, 10 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Queueing System
  3. Locality and Scalability
  4. Finite Queueing System
  5. Synchronous model
  6. Localized policies
  7. Symmetrized packet assignments
  8. Dynamical system model
  9. Sparse Graph Mean Field System
  10. Topological structure
  11. System model
  12. Optimality guarantees
  13. Reinforcement Learning
  14. Training on a finite queueing system
  15. Experiment Setup
  16. ...and 18 more sections

Key Result

theorem 1

Consider a sequence of finite graphs and initial states $(G_n, \mathbf z_n)$ converging in probability in the local weak sense to some limiting $(G, \mathbf z)$. For any policy $\pi$, as $n \to \infty$, we have convergence of the expected packet drop objective

Figures (10)

  • Figure 1: Visualization of how agents implement their policy. For instance, agent $i=2$ has neighbours $j\in\{1,3\}$. If its action is $a_2=0$, it will allocate all its arriving jobs to its own queue $j=2$ (green arrow). In contrast, if $a_2=1$ then the arriving jobs are allocated randomly to one of its neighbour $j$ (red arrows).
  • Figure 2: Performance comparison of the learned MF-R policy to NA-PS, JSQ, RND and OWN algorithms, on a CYC-1D graph, over a range of $\Delta t$s is shown, with $95\%$ confidence intervals depicted by error bars. The degree of each agent is $d=2$ and the number of agents (queues) used to make the graph are $N\in\{9, 21, 91, 901, 3501, 5001 \}$.
  • Figure 3: Performance of the MF-R policy for increasingly large CYC-1D graphs. The red horizontal line indicates the evaluated episode return of the learned MF-R policy during training on $N=101$, (MF-MFC). Shaded regions depict the $95\%$ confidence intervals.
  • Figure 4: Performance over $\Delta t\in\{1,2,\ldots, 10\}$ for large sparse graphs with underlying topologies of CCC in (a), TORUS in (b) and CM in (c). The number of nodes used to generate the graphs is given at the top of each subfigure.
  • Figure 5: Performance comparison on large-sized Bethe lattice graph. The MF-R can be worse than RND due to violation of modelling assumptions.
  • ...and 5 more figures

Theorems & Definitions (2)

  • theorem 1
  • corollary 1