Designing Skill-Compatible AI: Methodologies and Frameworks in Chess

TL;DR

The paper addresses how to design AI that is both high-performing and compatible with weaker partners in sequential decision tasks. It introduces two chess-focused frameworks, stochastic tag team (STT) and hand-and-brain (HB), and three senior strategies (Tree, Expector, Attuned) to study skill-compatibility under interruption and partner modeling. Across extensive experiments, the focal agents outperform a strong baseline Leela engine within these collaborative settings, demonstrating that compatibility is a distinct, measurable trait beyond raw strength; the authors also analyze mechanisms (tricking, helping, indirect effects) and cross-skill generalization to understand how compatibility arises. A code release and detailed methodology support replication and extension to other domains and human-in-the-loop studies.

Abstract

Powerful artificial intelligence systems are often used in settings where they must interact with agents that are computationally much weaker, for example when they work alongside humans or operate in complex environments where some tasks are handled by algorithms, heuristics, or other entities of varying computational power. For AI agents to successfully interact in these settings, however, achieving superhuman performance alone is not sufficient; they also need to account for suboptimal actions or idiosyncratic style from their less-skilled counterparts. We propose a formal evaluation framework for assessing the compatibility of near-optimal AI with interaction partners who may have much lower levels of skill; we use popular collaborative chess variants as model systems to study and develop AI agents that can successfully interact with lower-skill entities. Traditional chess engines designed to output near-optimal moves prove to be inadequate partners when paired with engines of various lower skill levels in this domain, as they are not designed to consider the presence of other agents. We contribute three methodologies to explicitly create skill-compatible AI agents in complex decision-making settings, and two chess game frameworks designed to foster collaboration between powerful AI agents and less-skilled partners. On these frameworks, our agents outperform state-of-the-art chess AI (based on AlphaZero) despite being weaker in conventional chess, demonstrating that skill-compatibility is a tangible trait that is qualitatively and measurably distinct from raw performance. Our evaluations further explore and clarify the mechanisms by which our agents achieve skill-compatibility.

Figures (5)

• Figure 1: Stochastic Tag Team Framework
• Figure 2: Hand and Brain Framework
• Figure 3: Ratio of probability of excess loss induction over different loss magnitudes
• Figure 4: $\Delta_{G_f}($maial, maiaf) as a function of different board win probabilities
• Figure 5: Agreement rate of different engines with each other