SMAC-R1: The Emergence of Intelligence in Decision-Making Tasks

TL;DR

SMAC-R1 presents a planner– coder–critic pipeline that leverages a large language model to generate interpretable decision-tree scripts for SMAC tasks. Through scripted planning, code generation, and environment-driven critique, followed by SFT, DPO, and GRPO, the approach achieves high win rates with minimal exploration and demonstrates strong transferability across similar SMAC maps. The work emphasizes white-box interpretability, map- and unit-informed strategies, and data-efficient fine-tuning, offering a scalable direction for domain-specific LLM training and decision-making pipelines. Overall, SMAC-R1 shows that combining code-generation and reinforcement-learning-inspired refinement can yield robust, transferable decision-making policies in complex multi-agent domains.

Abstract

StarCraft Multi-Agent Challenge (SMAC) has been one of the most commonly used experimental environments in multi-agent reinforcement learning (MARL), where the specific task is to control a set number of allied units to defeat enemy forces. Traditional MARL algorithms often require interacting with the environment for millions of steps to train a parametric model, of which the resulting policies are typically non-interpretable with weak transferability. In this paper, we introduce SMAC-R1 which is based on the Qwen2.5-7B-Base LLM distilled from DeepSeek-Coder-v2.5-236B. Similar to online reinforcement learning after behavior cloning in offline learning process, in our pipeline, agents leverage the DeepSeek LLM to generate decision tree code by providing task descriptions, and the agents are further self-reflected using feedback from the rewards provided by the environment. Based on that, we augment the generated scripts to fine-tune a small LLM, Qwen2.5-7B-Base, to distill the decision-making ability via Supervised Fine-Tuning (SFT) and enhance the script generation ability by the Group Relative Policy Optimization (GRPO) algorithm. We conduct experiments in the original 23 SMAC tasks and 10 newly-designed tasks to demonstrate that our method can produce high-quality, interpretable decision trees with minimal environmental exploration. Moreover, these scripts exhibit strong transferability, successfully applying to homogeneous SMAC environments without modification. We believe this approach offers a new direction for solving decision-making tasks and domain-specific LLM training pipelines in the future.

Figures (12)

• Figure 1: The overall architecture of our proposed LLM-SMAC framework. The framework takes the information of units, maps, and tasks as the environment prompts. Then, the planner generates a strategy conditioned on the environment, and the coder generates Python scripts following the strategy. The scripts are tested on the SMAC environment, and the results with the strategy and the coder are fed into the critic to analyze the refinement suggestion. Finally, the planner and the coder generate new strategies and codes in closed loops.
• Figure 2: The overall architecture of distilling, fine-tuning, and improving process of the student LLM (SMAC-R1). The generated decision-tree scripts are augmented by a coder LLM and generate a bunch of scripts with similar strategies. The scripts are divided into good, bad, and bug cases by evaluation on SMAC. Then the SMAC-R1 is fine-tuned by SFT, DPO, and finally GRPO by the data and iteratively fine-tuned by the scripts generated by our SMAC-R1 model.
• Figure 3: An example planner output that provides three potential skills: focus fire, hit and run, and Spread out and heal for the medivac unit. The planner also describes how to use each skill and when to use the skill.
• Figure 4: An example coder output from the first round and after the critic promotion round given a main strategy skeleton. Both of the two example codes follow the strategy, but the latter code is more precise.
• Figure 5: The learning curves among the 12/23 tasks in SMAC. The x-axis is the GRPO training steps and the y-axis is the average winning rate.(1/2)