Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles

TL;DR

This paper addresses the slow inference of diffusion-based game policies by introducing CPQE, a Consistency Policy with Q-Ensembles that enables fast, one-step action generation while maintaining strong performance. By combining a consistency model with an ensemble of Q-functions and a pessimistic lower-confidence bound, CPQE achieves up to 60 Hz inference and improved training stability compared to prior diffusion and single-Q baselines. The method demonstrates competitive rewards against multi-step diffusion policies across two Unity-based tasks, with clear gains in speed and stability, and shows that Q-ensembles yield more reliable value estimates than single Q-networks. The practical impact lies in enabling real-time, multi-modal policy learning for NPCs in games and other real-time applications where rapid inference and robust decision-making are critical.

Abstract

Diffusion models have shown impressive performance in capturing complex and multi-modal action distributions for game agents, but their slow inference speed prevents practical deployment in real-time game environments. While consistency models offer a promising approach for one-step generation, they often suffer from training instability and performance degradation when applied to policy learning. In this paper, we present CPQE (Consistency Policy with Q-Ensembles), which combines consistency models with Q-ensembles to address these challenges.CPQE leverages uncertainty estimation through Q-ensembles to provide more reliable value function approximations, resulting in better training stability and improved performance compared to classic double Q-network methods. Our extensive experiments across multiple game scenarios demonstrate that CPQE achieves inference speeds of up to 60 Hz -- a significant improvement over state-of-the-art diffusion policies that operate at only 20 Hz -- while maintaining comparable performance to multi-step diffusion approaches. CPQE consistently outperforms state-of-the-art consistency model approaches, showing both higher rewards and enhanced training stability throughout the learning process. These results indicate that CPQE offers a practical solution for deploying diffusion-based policies in games and other real-time applications where both multi-modal behavior modeling and rapid inference are critical requirements.

Figures (5)

• Figure 1: Inference time versus performance in Task 2 for each of the tested methods. CPQE provides a good trade-off between performance and inference time (in milliseconds). Diff-10, a diffusion policy trained with 10 denoising steps, reaches the highest reward but with the slowest inference time. CPQE outperforms the performance of Diff-5 while being $2\times$ faster.
• Figure 2: Overview of the environment used in this study. The environment is the same used by Sestini et al. sestini2023towards. The environment represents a 3D open- and procedurally generated-world. The agent can have multiple tasks in this environment, such as navigation or combat tasks. More details about the particular tasks used in this study are provided in Section \ref{['sec:env']}.
• Figure 3: Example of a trajectory in Task 2. The agent's starting position is on the ground, and it has to navigate to a elevator, wait for it to come down and jump over it. Once it is up on the building, the agent needs to cross a bridge between two high buildings: if it falls, there is no way to get back on track. Finally, the agent has to shoot at a destructible wall in order to reveal the goal location. This example is showing the CPQE policy acting in the environment.
• Figure 4: Comparison of inference times (in milliseconds) for each of the tested methods. CPBC is removed as it shares the same architecture and inference time as CPQL and CPQE. The figure shows the consistency-based methods as the methods with the lowest inference time.
• Figure 5: Training performance comparison of CPQL versus CPQE on Task 2 over 1,000 epochs. CPQE demonstrates both higher returns and enhanced training stability throughout the training process. The shaded area are the min and max over 3 seeds.