Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression

Xinming Gao, Shangzhe Li, Yujin Cai, Wenwu Yu

TL;DR

This work tackles the instability and dataset-specific hyperparameter tuning of offline Extreme Q-Learning (XQL) by introducing a quantile-regression-based, state-dependent temperature $β(s)$ and a mild generalization-based value regularization. By interpreting $V(s)$ and $\hat{V}(s)$ as quantiles of the $Q$-function and deriving $β(s)$ from the quantile gap via the Euler–Mascheroni constant $ω$, the authors formulate a $eta$-free learning objective (Quantile Q-Learning) with robust in-sample performance. They further stabilize training through a conservative regularization mechanism and a constrained policy objective, enabling strong results on D4RL and NeoRL2 with a single hyperparameter setting across domains. Experiments show competitive or superior performance relative to state-of-the-art baselines and confirm the stability advantages of QQL, with ablations highlighting the importance of value regulation and conservative estimation. The approach holds practical promise for real-world offline RL where domain-specific tuning is impractical and data collection is expensive or risky.

Abstract

Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme $Q$-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient $β$ via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.

Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression

TL;DR

This work tackles the instability and dataset-specific hyperparameter tuning of offline Extreme Q-Learning (XQL) by introducing a quantile-regression-based, state-dependent temperature and a mild generalization-based value regularization. By interpreting and as quantiles of the -function and deriving from the quantile gap via the Euler–Mascheroni constant , the authors formulate a -free learning objective (Quantile Q-Learning) with robust in-sample performance. They further stabilize training through a conservative regularization mechanism and a constrained policy objective, enabling strong results on D4RL and NeoRL2 with a single hyperparameter setting across domains. Experiments show competitive or superior performance relative to state-of-the-art baselines and confirm the stability advantages of QQL, with ablations highlighting the importance of value regulation and conservative estimation. The approach holds practical promise for real-world offline RL where domain-specific tuning is impractical and data collection is expensive or risky.

Abstract

Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme -Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.

Paper Structure

This paper contains 34 sections, 6 theorems, 28 equations, 4 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Extreme $Q$-Learning
  5. Revisiting Extreme $Q$-Learning with Quantile Regression
  6. Estimating $\beta$ under Mild Assumptions
  7. Remark on Assumption \ref{['ass: gumbel error 2']}
  8. Remark on Definition \ref{['def:soft-value']}
  9. Learning Value Functions with Quantile Regression
  10. Regulating Value Functions with Mild Generalization
  11. Experiments
  12. Experimental Setting and Baseline Methods
  13. Benchmark Datasets
  14. Baseline Methods
  15. Hyperparameter Setting
  16. ...and 19 more sections

Key Result

Proposition 1

Under Assumptions ass: gumbel error 1 and ass: gumbel error 2, given an optimal $Q$-function $Q^\star$ and $V^{\star}(s) = \max_aQ^{\star}(s,a)$, the following equation holds:

Figures (4)

  • Figure 1: Comparison of XQL performance using dataset-specific tuning versus consistent hyperparameters across the domain. In some scenarios, performance degrades notably without dataset-specific tuning.
  • Figure 2: QQL Performance and Ablation Analysis. We visualize the following results: (\ref{['fig:combined_QQL_vs_MXQL']}) Training stability comparison between QQL and MXQL. (\ref{['fig:q-mean']}) Q-value plots demonstrating the stabilization effect of value regularization and conservative estimation.
  • Figure 3: Ablation Studies on Value Regulation and Conservative Estimation Comparison of QQL performance to its variant without value regulation (w/o VR) and its variant without conservative estimation (w/o CE) adapted in the methodology section. The hyperparameters are consistent across variants.
  • Figure 4: Toy Example for $\beta(s)$ Scale. Empirical distributions of $-Q(s, a)$, along with the corresponding fitted Gumbel distributions and their estimated location, scale, and goodness-of-fit p-values.

Theorems & Definitions (18)

  • Proposition 1
  • proof
  • Definition 1
  • Proposition 2: Estimating $\beta(s)$
  • proof
  • Proposition 3
  • proof
  • Proposition 4
  • proof
  • Definition 2: Quantile Regression
  • ...and 8 more