Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy-Conditioned Policies for Multi-Agent Task Solving

Yue Lin, Shuhui Zhu, Wenhao Li, Ang Li, Dan Qiao, Pascal Poupart, Hongyuan Zha, Baoxiang Wang

TL;DR

The paper tackles non-stationarity in multi-agent learning by replacing opaque neural policies with executable, human-interpretable code and using LLMs as approximate interpreters. It reframes policy adaptation as learning a point-wise best-response operator, φ^i, via Programmatic Iterated Best Response (PIBR) that optimizes policy code against opponents’ code with textual gradients and unit-test feedback. Experiments on coordination matrix games and a cooperative Level-Based Foraging task show PIBR can reach cooperative equilibria in simple settings and demonstrate feasibility in more complex environments, albeit with variance and robustness challenges remaining. This work bridges program equilibrium theory and practical MARL, offering a novel pathway to policy conditioning and cooperative behavior through open, verifiable code syntheses guided by LLMs.

Abstract

In multi-agent tasks, the central challenge lies in the dynamic adaptation of strategies. However, directly conditioning on opponents' strategies is intractable in the prevalent deep reinforcement learning paradigm due to a fundamental ``representational bottleneck'': neural policies are opaque, high-dimensional parameter vectors that are incomprehensible to other agents. In this work, we propose a paradigm shift that bridges this gap by representing policies as human-interpretable source code and utilizing Large Language Models (LLMs) as approximate interpreters. This programmatic representation allows us to operationalize the game-theoretic concept of \textit{Program Equilibrium}. We reformulate the learning problem by utilizing LLMs to perform optimization directly in the space of programmatic policies. The LLM functions as a point-wise best-response operator that iteratively synthesizes and refines the ego agent's policy code to respond to the opponent's strategy. We formalize this process as \textit{Programmatic Iterated Best Response (PIBR)}, an algorithm where the policy code is optimized by textual gradients, using structured feedback derived from game utility and runtime unit tests. We demonstrate that this approach effectively solves several standard coordination matrix games and a cooperative Level-Based Foraging environment.

Policy-Conditioned Policies for Multi-Agent Task Solving

TL;DR

The paper tackles non-stationarity in multi-agent learning by replacing opaque neural policies with executable, human-interpretable code and using LLMs as approximate interpreters. It reframes policy adaptation as learning a point-wise best-response operator, φ^i, via Programmatic Iterated Best Response (PIBR) that optimizes policy code against opponents’ code with textual gradients and unit-test feedback. Experiments on coordination matrix games and a cooperative Level-Based Foraging task show PIBR can reach cooperative equilibria in simple settings and demonstrate feasibility in more complex environments, albeit with variance and robustness challenges remaining. This work bridges program equilibrium theory and practical MARL, offering a novel pathway to policy conditioning and cooperative behavior through open, verifiable code syntheses guided by LLMs.

Abstract

In multi-agent tasks, the central challenge lies in the dynamic adaptation of strategies. However, directly conditioning on opponents' strategies is intractable in the prevalent deep reinforcement learning paradigm due to a fundamental ``representational bottleneck'': neural policies are opaque, high-dimensional parameter vectors that are incomprehensible to other agents. In this work, we propose a paradigm shift that bridges this gap by representing policies as human-interpretable source code and utilizing Large Language Models (LLMs) as approximate interpreters. This programmatic representation allows us to operationalize the game-theoretic concept of \textit{Program Equilibrium}. We reformulate the learning problem by utilizing LLMs to perform optimization directly in the space of programmatic policies. The LLM functions as a point-wise best-response operator that iteratively synthesizes and refines the ego agent's policy code to respond to the opponent's strategy. We formalize this process as \textit{Programmatic Iterated Best Response (PIBR)}, an algorithm where the policy code is optimized by textual gradients, using structured feedback derived from game utility and runtime unit tests. We demonstrate that this approach effectively solves several standard coordination matrix games and a cooperative Level-Based Foraging environment.
Paper Structure (35 sections, 23 equations, 2 figures, 1 algorithm)

This paper contains 35 sections, 23 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Relation to Concurrent Work
  3. Preliminaries
  4. The Game
  5. The Solution Concepts
  6. Learning Methods for Markov Games
  7. Non-Stationarity
  8. Opponent Modeling
  9. Program Equilibrium and Provability Logic
  10. Method
  11. The Representational Bottleneck of Deep Reinforcement Learning
  12. A Paradigm Shift: Programmatic Policies by LLMs
  13. Refined Formalization: Meta-Programming and Mathematical Operators
  14. Learning
  15. Textual Loss Functions.
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Three $3 \times 3$ coordination matrix games used in our experiments. Each game defines payoffs for two agents (row agent's action $a^i \in \{0,1,2\}$, column agent's action $a^j \in \{0,1,2\}$): after they simultaneously choose a row and a column action, both receive the utility indicated by the selected matrix entry.
  • Figure 2: Experimental results.