Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach

TL;DR

The paper tackles the practical challenge of delivering human-like AI in games with limited training data by proposing a sample-efficient reinforcement learning framework built on Soft Actor-Critic (SAC). Key innovations include a replay-reset strategy, heavy offline data usage with symmetric sampling, and scenario-based curriculum learning, augmented by an expert-feedback loop for fine-tuning. In EA SPORTS FC 25, the approach trains a goalkeeper to exhibit proactivity and human-like positioning, outperforming the built-in AI in quantitative metrics and delivering smoother qualitative behavior, with training completed in under a day. The method also includes a robust evaluation framework, ablation studies, and demonstration of transfer to MuJoCo, underscoring practical applicability for production game development and potential adoption in ongoing game releases.

Abstract

While several high profile video games have served as testbeds for Deep Reinforcement Learning (DRL), this technique has rarely been employed by the game industry for crafting authentic AI behaviors. Previous research focuses on training super-human agents with large models, which is impractical for game studios with limited resources aiming for human-like agents. This paper proposes a sample-efficient DRL method tailored for training and fine-tuning agents in industrial settings such as the video game industry. Our method improves sample efficiency of value-based DRL by leveraging pre-collected data and increasing network plasticity. We evaluate our method training a goalkeeper agent in EA SPORTS FC 25, one of the best-selling football simulations today. Our agent outperforms the game's built-in AI by 10% in ball saving rate. Ablation studies show that our method trains agents 50% faster compared to standard DRL methods. Finally, qualitative evaluation from domain experts indicates that our approach creates more human-like gameplay compared to hand-crafted agents. As a testament to the impact of the approach, the method has been adopted for use in the most recent release of the series.

Figures (20)

• Figure 1: Main results of our approach. Top-left: our approach compared to removing the variations we add on top of the standard SAC algorithm. Bottom-left: training time compared to standard SAC algorithm. Our method outperforms built-in AI in less than one day. The standard SAC algorithm is able to match the performance of the built-in AI, but only after 4 days of training. Right: an example showcasing the behavioral differences between our agent (top) and the built-in AI (bottom) in the same situation. Our agent is more proactive and better understands the current situation, anticipating the shot. Quoting a professional goalkeeper: "the goalkeeper plays it really well! The keeper looks for opportunities to steal ground as the striker enters the box."
• Figure 2: Overview of the proposed method. (a) shows the agent composed of the policy and action-value functions as employed by SAC. (b) shows the main training framework, composed of the elements delineated in Section \ref{['sec:algorithm']}: curriculum learning, offline data, and network resets. (c) shows the three components of the evaluation framework: automatic quantitative evaluation, human qualitative evaluation, and expert-authored test suite, described in Section \ref{['sec:evaluation']}. Finally, (d) shows the fine tuning component. In this, we use only the failed scenario to generate new data but we combine it with the previously collected replay buffer. More details in Section \ref{['sec:fine_tuning']}.
• Figure 3: The impact of all the components of our method. Left: training curves comparing our approach removing different components. The curve represents the mean while the shaded areas represent the standard deviation of 5 different seeds. The drops in the plot correspond to network resets. Right: interquantile mean of the reward of all the ablations measured at the end of the training. As the plot shows, our agent achieves the highest performance in fewer training steps than the ablations, being more stable.
• Figure 4: The impact of symmetric sampling during fine-tuning. We compare fine-tuning using the symmetric sampling outlined in Section \ref{['sec:fine_tuning']} to a standard approach adding new data to the original buffer. Left: the success rate achieved by both agents in the failed scenario. Right: the episodic reward. Without symmetric sampling, the agent is not able to reach the same performance of our method in the same number of training steps.
• Figure 5: An example showcasing the behavioral differences between our agent (top) and the built-in AI (bottom) in the same situation. Our agent is more proactive and better understands the current situation, anticipating the shot and moving forward early. In contrast, the built-in AI is more passive, allowing the striker to score a goal by leaving the goal cage open. We report a quote from a professional goalkeeper: "the goalkeeper plays it really well! Keeper looks for opportunities to steal ground as the striker enters the box."