Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priors

TL;DR

This paper tackles data-inefficient exploration in deep reinforcement learning by integrating a control prior with an adaptive, context-aware weighting scheme. It introduces Contextualized Hybrid Ensemble Q-learning (CHEQ), which treats the blending weight as a context variable in a contextualized hybrid MDP and uses an ensemble of Q-functions to estimate epistemic uncertainty that guides weight adaptation. CHEQ leverages REDQ-style updates and a piecewise-linear mapping from uncertainty to a bounded RL contribution, enabling data-efficient learning and safer exploration without sacrificing eventual performance. Experiments on a car racing task show CHEQ achieves superior data efficiency, robustness to unknown scenarios, and strong zero-shot transfer compared to fixed-weight hybrids and state-of-the-art adaptive hybrids. The work provides a unified framework for analyzing hybrid RL and offers a practical algorithm that combines uncertainty-aware adaptation with ensemble-based acceleration.

Abstract

Combining Reinforcement Learning (RL) with a prior controller can yield the best out of two worlds: RL can solve complex nonlinear problems, while the control prior ensures safer exploration and speeds up training. Prior work largely blends both components with a fixed weight, neglecting that the RL agent's performance varies with the training progress and across regions in the state space. Therefore, we advocate for an adaptive strategy that dynamically adjusts the weighting based on the RL agent's current capabilities. We propose a new adaptive hybrid RL algorithm, Contextualized Hybrid Ensemble Q-learning (CHEQ). CHEQ combines three key ingredients: (i) a time-invariant formulation of the adaptive hybrid RL problem treating the adaptive weight as a context variable, (ii) a weight adaption mechanism based on the parametric uncertainty of a critic ensemble, and (iii) ensemble-based acceleration for data-efficient RL. Evaluating CHEQ on a car racing task reveals substantially stronger data efficiency, exploration safety, and transferability to unknown scenarios than state-of-the-art adaptive hybrid RL methods.

Figures (12)

• Figure 1: A standard RL setting (\ref{['fig:standard_rl']}), hybrid RL settings from prior work (\ref{['fig:nocontex_hybrid_rl']}) and our contextualized hybrid RL setting based on RL action $\mathbf{a}^\mathrm{RL}_t$ and weighting factor $\boldsymbol{\lambda}^\mathrm{RL}_t$ (\ref{['fig:contex_hybrid_rl']}).
• Figure 2: We illustrate different hybrid RL formulations on a cart pole system (\ref{['fig:cartpole_env']}) with a biased control prior, pushing to the left. Return of hybrid agents with fixed $\lambda^\mathrm{RL}_t$ (\ref{['fig:cartpole_lambda_fix_return']}) and variable $\lambda^\mathrm{RL}_t$ (\ref{['fig:cartpole_lambda_variable_return']}). Only the contextualized hybrid RL agent observing dynamics $\hat{p}$ can cope with both scenarios.
• Figure 3: Performance of all trained RL approaches in terms of evaluation return and training failures on the training track. Comparison of (\ref{['fig:cheq_vs_standard_rl_fixed']}) CHEQ with fixed-weight hybrid RL and standard RL, (\ref{['fig:cheq_vs_redq']}) increased UTD ratios and (\ref{['fig:cheq_vs_adaptive']}) prior work in adaptive hybrid RL.
• Figure 4: Comparison of return (\ref{['fig:transfer_boxplots']}) and failures (\ref{['fig:transfer_success_rates']}) of ten trained models per algorithm on ten transfer tracks. Development of $\lambda^\mathrm{RL}$ of the UTD-20 agent for an exemplary transfer track (\ref{['fig:uncertainty_map_transfer']}).
• Figure 5: Final return over number of fails during training (\ref{['fig:scatter_main_training']}) and zero-shot transfer (\ref{['fig:scatter_main_transfer']}).