Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents

TL;DR

This work uses Hanabi to probe cooperative reasoning in large language models, benchmarking 17 agents across 2–5 players under three prompting scaffolds to study the effects of context and memory on coordination. It introduces two public datasets, HanabiLogs and HanabiRewards, and demonstrates substantial gains from supervised fine-tuning and RL with verifiable rewards, achieving performance within a few points of frontier reasoning models and outperforming non-reasoning baselines on several tasks. The study reveals that explicit working-memory or step-by-step deductive scaffolds improve state-tracking and cross-play cooperation, though robust, human-level cooperative reasoning remains elusive. Beyond Hanabi, the RLVR-trained models show generalization to temporal reasoning and multi-agent coordination benchmarks, underscoring the broader potential of post-training cooperative reasoning for AI systems.

Abstract

Cooperative reasoning under incomplete information remains challenging for both humans and multi-agent systems. The card game Hanabi embodies this challenge, requiring theory-of-mind reasoning and strategic communication. We benchmark 17 state-of-the-art LLM agents in 2-5 player games and study the impact of context engineering across model scales (4B to 600B+) to understand persistent coordination failures and robustness to scaffolding: from a minimal prompt with only explicit card details (Watson setting), to scaffolding with programmatic, Bayesian-motivated deductions (Sherlock setting), to multi-turn state tracking via working memory (Mycroft setting). We show that (1) agents can maintain an internal working memory for state tracking and (2) cross-play performance between different LLMs smoothly interpolates with model strength. In the Sherlock setting, the strongest reasoning models exceed 15 points on average across player counts, yet still trail experienced humans and specialist Hanabi agents, both consistently scoring above 20. We release the first public Hanabi datasets with annotated trajectories and move utilities: (1) HanabiLogs, containing 1,520 full game logs for instruction tuning, and (2) HanabiRewards, containing 560 games with dense move-level value annotations for all candidate moves. Supervised and RL finetuning of a 4B open-weight model (Qwen3-Instruct) on our datasets improves cooperative Hanabi play by 21% and 156% respectively, bringing performance to within ~3 points of a strong proprietary reasoning model (o4-mini) and surpassing the best non-reasoning model (GPT-4.1) by 52%. The HanabiRewards RL-finetuned model further generalizes beyond Hanabi, improving performance on a cooperative group-guessing benchmark by 11%, temporal reasoning on EventQA by 6.4%, instruction-following on IFBench-800K by 1.7 Pass@10, and matching AIME 2025 mathematical reasoning Pass@10.

Figures (31)

• Figure 1: A comparison of the Watson: provides basic game state with only explicit knowledge and Sherlock settings, which scaffolds the agent's reasoning by providing belief state: listing all valid color/rank possibilities for every card in a "Deductive Context" block and enforcing a step-by-step Bayesian reasoning process for a 2-player Hanabi game.
• Figure 2: Average scores in 5 player Hanabi game for 10 runs of Grok-3-mini with different prompt strategies. Error bars are standard deviation.
• Figure 3: Average score of top-performing reasoning LLM based Hanabi agents when varying player count from 2 to 5. Error bars denote standard deviation.
• Figure 4: Scores of 17 state-of-the-art LLM Hanabi agents averaged over two to five-player settings. We show scores for each specific player count in both settings in Figure \ref{['fig:combined_hanabi_performance']} (Appendix \ref{['app:Hanabi_score']}). Error bars denote standard deviation.
• Figure 5: An example game state as viewed by Player 1 in the Mycroft setting. We note that each player has an independent deduction block for all other players hands. In this figure, we only show Player 1's deduction blocks for all other players.