When Are RL Hyperparameters Benign? A Study in Offline Goal-Conditioned RL

TL;DR

The paper addresses whether hyperparameter sensitivity in RL is intrinsic or amplified by training dynamics, by studying offline goal-conditioned RL with fixed data and controlled data-quality shifts. It contrasts bootstrapped TD learning (HIQL) with a non-TD quasimetric objective (QRL) and introduces inter-goal gradient alignment as a mechanistic diagnostic for sensitivity. Across stationary and non-stationary regimes, hyperparameter landscapes in offline GCRL are generally benign, but HIQL exhibits sharper optima and phase drift under data-quality schedules, while QRL retains broad near-optimal regions once modest expert data is present. The authors connect these differences to destructive inter-goal gradient interference in HIQL, suggesting that bootstrapping dynamics, rather than exploration alone, drive brittleness. Practically, the results imply that non-TD objectives may offer more robust hyperparameter behavior under distribution shifts, while TD-based methods may require adaptive stabilization or retuning across training phases.

Abstract

Hyperparameter sensitivity in Deep Reinforcement Learning (RL) is often accepted as unavoidable. However, it remains unclear whether it is intrinsic to the RL problem or exacerbated by specific training mechanisms. We investigate this question in offline goal-conditioned RL, where data distributions are fixed, and non-stationarity can be explicitly controlled via scheduled shifts in data quality. Additionally, we study varying data qualities under both stationary and non-stationary regimes, and cover two representative algorithms: HIQL (bootstrapped TD-learning) and QRL (quasimetric representation learning). Overall, we observe substantially greater robustness to changes in hyperparameter configurations than commonly reported for online RL, even under controlled non-stationarity. Once modest expert data is present ($\approx$ 20\%), QRL maintains broad, stable near-optimal regions, while HIQL exhibits sharp optima that drift significantly across training phases. To explain this divergence, we introduce an inter-goal gradient alignment diagnostic. We find that bootstrapped objectives exhibit stronger destructive gradient interference, which coincides directly with hyperparameter sensitivity. These results suggest that high sensitivity to changes in hyperparameter configurations during training is not inevitable in RL, but is amplified by the dynamics of bootstrapping, offering a pathway toward more robust algorithmic objective design.

Figures (19)

• Figure 1: Isolation of objective sensitivity to hyperparameter configuration changes. By using offline GCRL, we isolate how learning mechanisms respond to data quality without the confound of exploration noise. We visualize the near-optimal region (top-10%) in the learning-rate -- discount-factor plane (HIQL on the left and QRL on the right). Top: Scheduled training phases ($1 \to 4$) with improving data quality. Bottom: Fixed offline mixtures at different quality levels. Even when data is fixed (offline), the bootstrapped method (HIQL) exhibits significant landscape drift, whereas the representation learning method (QRL) remains stable.
• Figure 2: Phase-wise concentration and stability of fANOVA importances. We compare hyperparameter influence across scheduled data quality phases. Left Column (a, c): Perplexity of the normalized importance distribution; lower values indicate importance concentrates on fewer hyperparameters. Right Column (b, d): Importance-weighted Kendall's $\tau$ between consecutive phases; higher values indicate a more stable ordering of influential hyperparameters. Results are shown for antmaze-large-v0 (top) and antmaze-medium-v0 (bottom).
• Figure 3: Inter-goal gradient alignment. Empirical CDF of cosine similarity $\kappa(g,g')$ between gradients induced by different goal relabelings. More negative similarities indicate stronger inter-goal interference. HIQL exhibits substantially more negative mass than QRL, especially in the critic updates, suggesting that bootstrapped targets being associated with sensitivity to the hyperparameters studied.
• Figure 4: Phased training mohan-automlconf23a. At $t_0$ we start training all sampled configurations and evaluate them at $t_{ls(1)}$. For the second phase, we start training from $t_{ls(1)}$, loading the checkpoint of the best configuration from phase one, and then re-evaluate all configurations at $t_{ls(2)}$. The best configuration is determined by evaluating at $t_{\text{final}}$, which we will set to $t_{ls(i)}$.
• Figure 5: QRL: Phase-wise fANOVA main-effect importances under scheduled data quality for antmaze-medium-v0.