Online Decision-Making in Tree-Like Multi-Agent Games with Transfers

TL;DR

This work tackles online decision-making in tree-structured multi-agent games where actions propagate upward and agents are self-interested. It introduces MAIL, a decentralized algorithm that first learns optimal incentives (transfers) and then optimizes actions via a shifted-bandit subroutine, enabling no-regret learning across the hierarchy. The authors prove that, under depth-dependent parameter choices, each node achieves regret $\tilde{O}(T^{1-1/(2d^2)})$ with high probability and $o(T)$ in expectation, and the overall social welfare regret is also $o(T)$, implying convergence to a social-welfare optimal, subgame-perfect Nash equilibrium without a central mediator. The results demonstrate that transfers can align hierarchical agents to behave as if they were collaborating, with practical implications for decentralized incentive design in cascaded ML systems and other nested principal–agent settings.

Abstract

The widespread deployment of Machine Learning systems everywhere raises challenges, such as dealing with interactions or competition between multiple learners. In that goal, we study multi-agent sequential decision-making by considering principal-agent interactions in a tree structure. In this problem, the reward of a player is influenced by the actions of her children, who are all self-interested and non-cooperative, hence the complexity of making good decisions. Our main finding is that it is possible to steer all the players towards the globally optimal set of actions by simply allowing single-step transfers between them. A transfer is established between a principal and one of her agents: the principal actually offers the proposed payment if the agent picks the recommended action. The analysis poses specific challenges due to the intricate interactions between the nodes of the tree and the propagation of the regret within this tree. Considering a bandit setup, we propose algorithmic solutions for the players to end up being no-regret with respect to the optimal pair of actions and incentives. In the long run, allowing transfers between players makes them act as if they were collaborating together, although they remain self-interested non-cooperative: transfers restore efficiency.

Figures (2)

• Figure 1: Illustration of the game structure: ...
• Figure 2: Cumulative regret for different algorithms on a $5$ arms instance.

Theorems & Definitions (35)

• Lemma 4.0
• Lemma 4.0
• Lemma 4.0
• Corollary 4.0
• Lemma 4.0
• Theorem 4.1
• Corollary 4.1
• Lemma 5.0
• proof : Proof of \ref{['lemma:optimal_incentives']}
• Lemma C.0