Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization

TL;DR

This work tackles estimation bias in deterministic policy gradient methods for continuous control by extending TD error-driven regularization (TDDR) with two refinements: a tunable bias mechanism governed by a single hyperparameter and a dynamics-based representation-learning module. It introduces three core variants—DADC, DASC, and SASC—and their representation-enhanced forms (DADC-R, DASC-R, SASC-R), employing a TD-error-driven convex combination to balance pessimistic and optimistic estimates. The representation learning component augments state and state–action inputs with learned embeddings, improving stability and performance. Comprehensive experiments on MuJoCo and Box2D show substantial gains over baselines and competitive SOTA performance, with evidence that the optimal bias strategy is task-dependent and that both overestimation and underestimation can be advantageous depending on the environment.

Abstract

Deterministic policy gradient algorithms for continuous control suffer from value estimation biases that degrade performance. While double critics reduce such biases, the exploration potential of double actors remains underexplored. Building on temporal-difference error-driven regularization (TDDR), a double actor-critic framework, this work introduces enhanced methods to achieve flexible bias control and stronger representation learning. We propose three convex combination strategies, symmetric and asymmetric, that balance pessimistic estimates to mitigate overestimation and optimistic exploration via double actors to alleviate underestimation. A single hyperparameter governs this mechanism, enabling tunable control across the bias spectrum. To further improve performance, we integrate augmented state and action representations into the actor and critic networks. Extensive experiments show that our approach consistently outperforms benchmarks, demonstrating the value of tunable bias and revealing that both overestimation and underestimation can be exploited differently depending on the environment.

Figures (5)

• Figure 1: Architecture of DADC-R, DASC-R, and SASC-R: $q_{ij} = Q_{\theta_i^\prime}([s^\prime,e_{s^\prime}],a_j^\prime,e_{s^\prime a_j^\prime})$ and $q_i = Q_{\theta_i^\prime}([s,e_s],a,e_{sa})$; the duplication of $Q_{\theta_1^\prime}/Q_{\theta_2^\prime}$ indicates that the same networks are used with different inputs; the action $a_i'$ is generated by $\pi_{\phi_i^\prime}([s^{\prime},e_{s^\prime}])$; the notation $[\cdot]$ represents the concatenation of vectors along the feature dimension; "min" means a minimum of two values. The input $\psi$ is partitioned into two components, with $\upsilon$ representing the common part across all algorithms, and $1-\upsilon$ corresponding to the algorithm-specific part. When $|\delta_1| \leq |\delta_2|$, DADC-R combines $\upsilon\min(q_{11}, q_{21})$ and $(1-\upsilon) \min(q_{12}, q_{22})$. Conversely, when $|\delta_1| > |\delta_2|$, DADC-R combines $\upsilon \min(q_{12}, q_{22})$ and $(1-\upsilon) \min(q_{11}, q_{21})$. DASC-R and SASC-R adhere to the same computational rules.
• Figure 2: Comparison of DADC, DASC, SASC, DADC-R, DASC-R, SASC-R, TD3 and TDDR across four environments. (a) Ant-v2, (b) HalfCheetah-v2, (c) Hopper-v2, (d) Walker2d-v2.
• Figure 3: Comparison of estimation bias for DADC, DASC, and SASC across four environments. Markers denote distinct environments: diamonds for Ant, squares for HalfCheetah, triangles for Hopper, and circles for Walker2d.
• Figure 4: Comparison of estimation bias among DADC, DASC, SASC, TD3, and TDDR across four environments. (a) Ant-v2, (b) HalfCheetah-v2, (c) Hopper-v2, (d) Walker2d-v2.
• Figure 5: Box plot illustrating the minimum, maximum, mean, and variance of the nine normalized average returns from the comparisons in Tables \ref{['tab:numerical_comparison_baseline_algorithm']}, \ref{['tab:numerical_comparison_algorithm']}, and \ref{['table.SOTA']}. Red diamonds, orange squares, green triangles, and blue circles denote performance at 300k steps in Ant, HalfCheetah, Hopper, and Walker2d, respectively.

Theorems & Definitions (1)

• Remark 1