Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning for Game-Theoretic Resource Allocation on Graphs

Zijian An, Lifeng Zhou

TL;DR

This work tackles GRAG, a multi-step, graph-constrained resource allocation problem modeled as an on-graph Colonel Blotto game and framed as an MDP suitable for reinforcement learning. It introduces an action-displacement adjacency matrix to restrict actions to graph-feasible moves, enabling reliable learning with DQN and PPO despite a dynamically changing action space. Across symmetric and asymmetric graphs, the RL agents outperform baselines, with DQN showing stronger generalization and PPO exhibiting more variable performance; asymmetric graphs reveal the agents' ability to exploit structural advantages to secure favorable outcomes. The results demonstrate the viability of RL for complex, graph-structured strategic allocation and provide a foundation for extending to weighted, unequal, or continuous resources in future work.

Abstract

Game-theoretic resource allocation on graphs (GRAG) involves two players competing over multiple steps to control nodes of interest on a graph, a problem modeled as a multi-step Colonel Blotto Game (MCBG). Finding optimal strategies is challenging due to the dynamic action space and structural constraints imposed by the graph. To address this, we formulate the MCBG as a Markov Decision Process (MDP) and apply Reinforcement Learning (RL) methods, specifically Deep Q-Network (DQN) and Proximal Policy Optimization (PPO). To enforce graph constraints, we introduce an action-displacement adjacency matrix that dynamically generates valid action sets at each step. We evaluate RL performance across a variety of graph structures and initial resource distributions, comparing against random, greedy, and learned RL policies. Experimental results show that both DQN and PPO consistently outperform baseline strategies and converge to a balanced $50\%$ win rate when competing against the learned RL policy. Particularly, on asymmetric graphs, RL agents successfully exploit structural advantages and adapt their allocation strategies, even under disadvantageous initial resource distributions.

Reinforcement Learning for Game-Theoretic Resource Allocation on Graphs

TL;DR

This work tackles GRAG, a multi-step, graph-constrained resource allocation problem modeled as an on-graph Colonel Blotto game and framed as an MDP suitable for reinforcement learning. It introduces an action-displacement adjacency matrix to restrict actions to graph-feasible moves, enabling reliable learning with DQN and PPO despite a dynamically changing action space. Across symmetric and asymmetric graphs, the RL agents outperform baselines, with DQN showing stronger generalization and PPO exhibiting more variable performance; asymmetric graphs reveal the agents' ability to exploit structural advantages to secure favorable outcomes. The results demonstrate the viability of RL for complex, graph-structured strategic allocation and provide a foundation for extending to weighted, unequal, or continuous resources in future work.

Abstract

Game-theoretic resource allocation on graphs (GRAG) involves two players competing over multiple steps to control nodes of interest on a graph, a problem modeled as a multi-step Colonel Blotto Game (MCBG). Finding optimal strategies is challenging due to the dynamic action space and structural constraints imposed by the graph. To address this, we formulate the MCBG as a Markov Decision Process (MDP) and apply Reinforcement Learning (RL) methods, specifically Deep Q-Network (DQN) and Proximal Policy Optimization (PPO). To enforce graph constraints, we introduce an action-displacement adjacency matrix that dynamically generates valid action sets at each step. We evaluate RL performance across a variety of graph structures and initial resource distributions, comparing against random, greedy, and learned RL policies. Experimental results show that both DQN and PPO consistently outperform baseline strategies and converge to a balanced win rate when competing against the learned RL policy. Particularly, on asymmetric graphs, RL agents successfully exploit structural advantages and adapt their allocation strategies, even under disadvantageous initial resource distributions.
Paper Structure (17 sections, 3 theorems, 14 equations, 7 figures, 5 tables)

This paper contains 17 sections, 3 theorems, 14 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. On-Graph Colonel Blotto Game
  4. Reinforcement Learning for GRAG
  5. Approach
  6. RL Conventions
  7. Valid Action Set
  8. Action-displacement adjacency matrix with different resources
  9. DQN Framework
  10. PPO Framework
  11. Experiments
  12. RL agent against random-policy agent on $G_0$ and $G_1$
  13. RL agent against greedy-policy agent on $G_0$ and $G_1$
  14. RL agent against RL agent on $G_0$ and $G_1$
  15. RL agent against random-policy agent on $G_0$ with different amount of resources
  16. ...and 2 more sections

Key Result

Theorem 1

For any resource $j$ on node $n_j$ at time step $t$, $j\in\mathcal{I}(M)$ and $n_j\in\mathcal{I}(N)$, $\mathbf{a}_j\in\mathcal{K}:=\{k\in\mathcal{I}(N)\ |\ \mathbf{J}_{jk}=1\}$ is valid resource action.

Figures (7)

  • Figure 1: GRAG on a five-node graph involves two players, the red and blue, each with eight units of resources. The players make moves simultaneously. In the left subfigure, the game begins in a tied status. The red has a clear winning strategy: moving a resource from node 1 to node 5. However, the blue player is aware of this strategy and can counter it by moving one resource from node 5 to node 1. As shown in the right subfigure, this counteraction restores a tie, and the game continues and evolves to the next step.
  • Figure 2: Game flow and DQN process with Player 1 as the RL agent.
  • Figure 3: Game flow and PPO network with Player 1 as the RL agent. The game flow structure is the same as in Figure \ref{['fig:illus_2']}.
  • Figure 4: Five graphs $G_0, G_1, G_2, G_3, G_4$, where a node is indexed numerically and represents a discrete location for resource allocation. Directed arrows between nodes indicate one-way connectivity, specifying the allowable transitions of resources between locations.
  • Figure 5: Illustration of four distinct initial resource distributions $C_1$, $C_2$, $C_3$ and $C_4$ on graphs $G_0$ (\ref{['g0']}) and $G_1$ (\ref{['g1']}).
  • ...and 2 more figures

Theorems & Definitions (5)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Corollary 1