KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination

TL;DR

This work tackles zero-shot coordination by moving beyond black-box policies toward interpretable, knowledge-driven programs. It introduces KnowPC, a framework that learns policies as symbolic DSL programs by extracting environment transition knowledge with an extractor and reasoning about preconditions with a reasoner, followed by genetics-based program synthesis. Evaluations on Overcooked across multiple layouts and a ZSC+ setting show that KnowPC outperforms deep RL baselines in ZSC, generalizes to layout changes, and maintains interpretability through explicit program logic. The approach offers robust coordination with unseen partners and humans, especially under sparse rewards, with potential for integration of larger models to bootstrap primitives in future work; the program search space is vast (approximately $1.64\times10^{39}$ candidates), underscoring the value of guided synthesis.

Abstract

Zero-shot coordination (ZSC) remains a major challenge in the cooperative AI field, which aims to learn an agent to cooperate with an unseen partner in training environments or even novel environments. In recent years, a popular ZSC solution paradigm has been deep reinforcement learning (DRL) combined with advanced self-play or population-based methods to enhance the neural policy's ability to handle unseen partners. Despite some success, these approaches usually rely on black-box neural networks as the policy function. However, neural networks typically lack interpretability and logic, making the learned policies difficult for partners (e.g., humans) to understand and limiting their generalization ability. These shortcomings hinder the application of reinforcement learning methods in diverse cooperative scenarios.We suggest to represent the agent's policy with an interpretable program. Unlike neural networks, programs contain stable logic, but they are non-differentiable and difficult to optimize.To automatically learn such programs, we introduce Knowledge-driven Programmatic reinforcement learning for zero-shot Coordination (KnowPC). We first define a foundational Domain-Specific Language (DSL), including program structures, conditional primitives, and action primitives. A significant challenge is the vast program search space, making it difficult to find high-performing programs efficiently. To address this, KnowPC integrates an extractor and an reasoner. The extractor discovers environmental transition knowledge from multi-agent interaction trajectories, while the reasoner deduces the preconditions of each action primitive based on the transition knowledge.

Figures (13)

• Figure 1: Illustration of Overcooked environment. We choose two layouts Forced Coordination and Asymmetric Advantages for demonstration.
• Figure 2: The overall framework of KnowPC. The data buffer is maintained and continues to increase in the learning loop.
• Figure 3: The domain-specific language for constructing our programs. IT is a module that contain if-then structure. $A$ and $B$ are action primitive and condition primitive. $\mathcal{B}$ is a conjunction of conditions. An element of $E$ is IT.
• Figure 4: Definitions of condition primitive $A$ and action primitive $B$. A vertical bar $|$ indicates choice.
• Figure 5: Illustration of a simple transition.

Theorems & Definitions (2)

• Definition 1
• Definition 2