Hypernetwork-based approach for optimal composition design in partially controlled multi-agent systems

TL;DR

This work addresses the composition design problem in Partially Controlled Multi-Agent Systems (PCMAS) by formulating it as a bi-level optimization over the number of controllable agents $N_c$ and their policies, while uncontrollable agents follow best-response behaviors. It introduces a novel hypernetwork-based framework that jointly optimizes system composition and agent policies, enabling unified policy generation across all configurations and reducing re-learning overhead. The approach augments with reward-parameter optimization and a mean action network to improve scalability and coordination, and demonstrates near-equilibrium policy approximation and substantial system-performance gains on NYC taxi data, including improvements up to $13.89\%$ in the objective function. The findings offer practical insights into when controllable agents are most beneficial and showcase the framework’s potential to enhance decision-making in large-scale PCMAS applications.

Abstract

Partially Controlled Multi-Agent Systems (PCMAS) are comprised of controllable agents, managed by a system designer, and uncontrollable agents, operating autonomously. This study addresses an optimal composition design problem in PCMAS, which involves the system designer's problem, determining the optimal number and policies of controllable agents, and the uncontrollable agents' problem, identifying their best-response policies. Solving this bi-level optimization problem is computationally intensive, as it requires repeatedly solving multi-agent reinforcement learning problems under various compositions for both types of agents. To address these challenges, we propose a novel hypernetwork-based framework that jointly optimizes the system's composition and agent policies. Unlike traditional methods that train separate policy networks for each composition, the proposed framework generates policies for both controllable and uncontrollable agents through a unified hypernetwork. This approach enables efficient information sharing across similar configurations, thereby reducing computational overhead. Additional improvements are achieved by incorporating reward parameter optimization and mean action networks. Using real-world New York City taxi data, we demonstrate that our framework outperforms existing methods in approximating equilibrium policies. Our experimental results show significant improvements in key performance metrics, such as order response rate and served demand, highlighting the practical utility of controlling agents and their potential to enhance decision-making in PCMAS.

Figures (13)

• Figure 1: Hypernetwork-based architecture for solving the composition design problem. Policies for controllable ($c$-agents) and uncontrollable ($u$-agents) agents are generated by hypernetworks based on $N_c$. The system composition and environmental state are concatenated (denoted by $\oplus$) and used as input to the target network.
• Figure 2: The overall architecture of the proposed framework. The dotted lines represent the network update process.
• Figure 3: Geographical representation of the study area, covering zones from Manhattan to LaGuardia Airport.
• Figure 4: Comparison of NashConv values across our algorithm and baselines when $\alpha=0.00$.
• Figure 5: Objective value improvements for varying hourly rates with $k=0.6$.