SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks

TL;DR

This paper tackles the instability and overestimation challenges of offline reinforcement learning by proposing EDTD7, a model-free actor-critic that integrates State-Action Learned Embeddings (SALE) with ensemble Q-networks and a gradient-diversity penalty. It introduces a comprehensive loss framework, including a Q-value loss with a stabilized minimum over the ensemble, a policy loss with adaptive behavior cloning, a SALE encoder loss, and LAP-based prioritized replay, all designed to maintain training stability. Empirical results on the D4RL MuJoCo suite show EDTD7 achieves strong convergence speed, stability, and competitive performance relative to TD7, EDAC, and other baselines, with insightful ablations validating the contributions of SALE, LAP, and the ensemble design. The work highlights the practicality of a simple, ensemble-based offline RL approach that balances stability and efficiency, and suggests future work on automating hyperparameter selection and exploring deeper interactions between ensemble methods and dataset structure.

Abstract

In this work, we build upon the offline reinforcement learning algorithm TD7, which incorporates State-Action Learned Embeddings (SALE) and a prioritized experience replay buffer (LAP). We propose a model-free actor-critic algorithm that integrates ensemble Q-networks and a gradient diversity penalty from EDAC. The ensemble Q-networks introduce penalties to guide the actor network toward in-distribution actions, effectively addressing the challenge of out-of-distribution actions. Meanwhile, the gradient diversity penalty encourages diverse Q-value gradients, further suppressing overestimation for out-of-distribution actions. Additionally, our method retains an adjustable behavior cloning (BC) term that directs the actor network toward dataset actions during early training stages, while gradually reducing its influence as the precision of the Q-ensemble improves. These enhancements work synergistically to improve the stability and precision of the training. Experimental results on the D4RL MuJoCo benchmarks demonstrate that our algorithm achieves higher convergence speed, stability, and performance compared to existing methods.

Figures (7)

• Figure 1: Overall network architecture. The State-Action Learned Embeddings (SALE) method uses encoders $\left( f_{\theta_s}, g_{\theta_{sa}} \right)$ to output the embeddings $\left( z^s, z^{sa} \right)$ for states and state-action pairs, respectively. These embeddings are then passed through the corresponding hidden layers of the ensemble Q-network $Q_{\theta_i}$ and the policy network $\pi_{\varphi}$, where $h_i$ denotes the output of the $i$-th hidden layer, and $d(\cdot)$ represents the dimension of the input object (such as the state $s$ or other objects). A linear transformation $L(d(\cdot), d(\cdot))$ is used to map the input dimensions to output dimensions in each hidden layer. After passing through the respective activation functions, the ensemble Q-network outputs the minimum value among the ensemble Q-values, while the policy network $\pi_{\varphi}$ generates an action in the range of (-1, 1) after being processed by the $\tanh$ activation function.
• Figure 2: The learning curves illustrate the performance of EDTD7 in comparison to TD7, EDAC, IQL, and CQL. Each curve represents the average performance over 4 random seeds, with the shaded regions indicating the standard deviation across these seeds.
• Figure 3: The learning curves illustrate the performance of EDTD7 with varying ensemble sizes. Each curve represents the average performance, and the shaded regions indicate the standard deviation.
• Figure 4: The learning curves illustrate the impact of the $\eta$ term (Q-value gradient diversity penalty) in EDTD7. Each curve represents the average performance for different values of $\eta$, where a larger $\eta$ indicates a stronger penalty and higher gradient diversity. The shaded regions indicate the standard deviation.
• Figure 5: The learning curves illustrate the impact of the $\lambda$ term (imitation learning strength) in EDTD7. Each curve represents the average performance for different values of $\lambda$, where a larger $\lambda$ indicates a stronger emphasis on imitation learning. The shaded regions indicate the standard deviation.