CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space

TL;DR

Hybrid action spaces challenge deep RL due to limited policy expressiveness and scalability. CHDP addresses this by deploying two cooperative diffusion policies for discrete and continuous actions, coupled via a sequential update scheme and a Q-guided codebook that embeds high-dimensional discrete actions into a compact latent space. The framework achieves state-of-the-art results across eight PAMDP benchmarks, with up to 19.3% gains in final success rate, and qualitative analyses reveal multi-modal, coordinated strategies that traditional approaches miss. This scalable, end-to-end approach is well-suited for real-world domains like robotics and game AI where actions comprise discrete choices plus continuous parameters.

Abstract

Hybrid action space, which combines discrete choices and continuous parameters, is prevalent in domains such as robot control and game AI. However, efficiently modeling and optimizing hybrid discrete-continuous action space remains a fundamental challenge, mainly due to limited policy expressiveness and poor scalability in high-dimensional settings. To address this challenge, we view the hybrid action space problem as a fully cooperative game and propose a \textbf{Cooperative Hybrid Diffusion Policies (CHDP)} framework to solve it. CHDP employs two cooperative agents that leverage a discrete and a continuous diffusion policy, respectively. The continuous policy is conditioned on the discrete action's representation, explicitly modeling the dependency between them. This cooperative design allows the diffusion policies to leverage their expressiveness to capture complex distributions in their respective action spaces. To mitigate the update conflicts arising from simultaneous policy updates in this cooperative setting, we employ a sequential update scheme that fosters co-adaptation. Moreover, to improve scalability when learning in high-dimensional discrete action space, we construct a codebook that embeds the action space into a low-dimensional latent space. This mapping enables the discrete policy to learn in a compact, structured space. Finally, we design a Q-function-based guidance mechanism to align the codebook's embeddings with the discrete policy's representation during training. On challenging hybrid action benchmarks, CHDP outperforms the state-of-the-art method by up to $19.3\%$ in success rate.

Figures (4)

• Figure 1: An overview of our CHDP framework. (Left) Inference: The discrete policy's latent representation is quantized by a codebook to yield a discrete action and relevant codeword. This codeword conditions the continuous policy for generating the final continuous action. (Right) Training: Guided by a shared Q-function, the discrete policy is updated first. Subsequently, conditioned on the output of this updated discrete policy, the continuous policy and the codebook are jointly optimized.
• Figure 2: The process of aligning latent representations and codewords.
• Figure 3: Visualizations of the tested environments.
• Figure 4: Comparisons of algorithms in different environments. The x- and y-axis represent the environment steps ($\times 10^5$) and the average success rate, respectively. The solid curve and the shaded region denote the mean and standard deviation over 5 independent runs. All curves are smoothed for visual clarity.