Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computational Intractability of Strategizing against Online Learners

Angelos Assos, Yuval Dagan, Nived Rajaraman

TL;DR

This work addresses the problem of how an optimizer should strategize against online no-regret learners in general two-player repeated games. It establishes a strong NP-hardness result: unless $P=NP$, no polynomial-time optimizer can achieve a near-optimal strategy against Hedge/MWU with horizon $T$, even when action spaces grow only poly$(T)$ and the learner uses a slow-learning rate $\\eta=1/T^{1-\\alpha}$. The authors prove an $\\Omega(T)$ additive hardness by a reduction from $(1,2)$-maxTSP, using a carefully constructed game with initialization and a no-init variant, and showing that any near-optimal policy would yield a good maxTSP solution, contradicting its known hardness. This result reveals a fundamental computational barrier to optimizing against learning agents in general games, suggesting that efficient optimizers may only exist for highly structured settings. The work also delineates how prior weaker hardness results from fictitious-play-type learners do not extend to standard no-regret learners, strengthening the case for structure-aware approaches in practice.

Abstract

Online learning algorithms are widely used in strategic multi-agent settings, including repeated auctions, contract design, and pricing competitions, where agents adapt their strategies over time. A key question in such environments is how an optimizing agent can best respond to a learning agent to improve its own long-term outcomes. While prior work has developed efficient algorithms for the optimizer in special cases - such as structured auction settings or contract design - no general efficient algorithm is known. In this paper, we establish a strong computational hardness result: unless $\mathsf{P} = \mathsf{NP}$, no polynomial-time optimizer can compute a near-optimal strategy against a learner using a standard no-regret algorithm, specifically Multiplicative Weights Update (MWU). Our result proves an $Ω(T)$ hardness bound, significantly strengthening previous work that only showed an additive $Θ(1)$ impossibility result. Furthermore, while the prior hardness result focused on learners using fictitious play - an algorithm that is not no-regret - we prove intractability for a widely used no-regret learning algorithm. This establishes a fundamental computational barrier to finding optimal strategies in general game-theoretic settings.

Computational Intractability of Strategizing against Online Learners

TL;DR

This work addresses the problem of how an optimizer should strategize against online no-regret learners in general two-player repeated games. It establishes a strong NP-hardness result: unless , no polynomial-time optimizer can achieve a near-optimal strategy against Hedge/MWU with horizon , even when action spaces grow only poly and the learner uses a slow-learning rate . The authors prove an additive hardness by a reduction from -maxTSP, using a carefully constructed game with initialization and a no-init variant, and showing that any near-optimal policy would yield a good maxTSP solution, contradicting its known hardness. This result reveals a fundamental computational barrier to optimizing against learning agents in general games, suggesting that efficient optimizers may only exist for highly structured settings. The work also delineates how prior weaker hardness results from fictitious-play-type learners do not extend to standard no-regret learners, strengthening the case for structure-aware approaches in practice.

Abstract

Online learning algorithms are widely used in strategic multi-agent settings, including repeated auctions, contract design, and pricing competitions, where agents adapt their strategies over time. A key question in such environments is how an optimizing agent can best respond to a learning agent to improve its own long-term outcomes. While prior work has developed efficient algorithms for the optimizer in special cases - such as structured auction settings or contract design - no general efficient algorithm is known. In this paper, we establish a strong computational hardness result: unless , no polynomial-time optimizer can compute a near-optimal strategy against a learner using a standard no-regret algorithm, specifically Multiplicative Weights Update (MWU). Our result proves an hardness bound, significantly strengthening previous work that only showed an additive impossibility result. Furthermore, while the prior hardness result focused on learners using fictitious play - an algorithm that is not no-regret - we prove intractability for a widely used no-regret learning algorithm. This establishes a fundamental computational barrier to finding optimal strategies in general game-theoretic settings.

Paper Structure

This paper contains 34 sections, 19 theorems, 81 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Related work
  4. Multi-Agent Learning and Strategic Optimization
  5. Optimizing Against Learning Agents in Structured Settings
  6. General Games: Lack of Efficient Optimizer Algorithms
  7. Computational Hardness in Special Cases
  8. Learning the Stackelberg Equilibrium from Repeated Interactions
  9. Preliminaries
  10. Reduction Workhorse: $(1,2)$-TSP
  11. Lower bound against MWU with non-zero reward history
  12. Structure of the reduction
  13. Proof of \ref{['theorem:with_initialization']}: non-zero historical rewards
  14. Proof sketch of \ref{['theorem:with_initialization']}.
  15. Full proof of \ref{['theorem:with_initialization']}.
  16. ...and 19 more sections

Key Result

Theorem 1.1

Fix absolute constants $\alpha, \beta \in (0,1]$. Let $T$ denote the horizon, suppose the learner plays Hedge with learning rate parameterized as $\eta = 1/T^{1-\alpha}$ (cf. def:MWU) and suppose that the number of strategies per player is bounded by $|\mathcal{A}|, |\mathcal{B}| \leqslant T^\beta$. for an absolute constant $c > 0$.

Theorems & Definitions (31)

  • Theorem 1.1
  • Definition 2.1: Hedge/MWU learner
  • Definition 2.2: Optimizing against MWU
  • Definition 2.3: $(1,2)$-TSP
  • Theorem 2.4: Hardness of $(1,2)$-TSP karpinski2012approximation
  • Definition 2.5: $(1,2)$-maxTSP
  • Theorem 2.6
  • proof
  • Theorem 3.1
  • Definition 3.2
  • ...and 21 more