Your Policy Regularizer is Secretly an Adversary

TL;DR

This work reframes policy regularization in reinforcement learning as robustness to adversarial reward perturbations and uses convex duality to derive the robust set of perturbations under $KL$ and $α$-divergences. It shows that the worst-case reward perturbations correspond to the gradient of the regularizer and that the policy-form and value-form perturbations coincide at optimality, linked through path-consistency and an indifference condition. The authors provide a detailed theoretical development plus visual experiments illustrating the robust set, worst-case perturbations, and certificates of optimality, and they contrast divergence-based regularization with entropy-based analyses. The results clarify how regularized policies generalize to adversarially perturbed rewards and connect to related algorithms that leverage duality and path-consistency in learning objectives.

Abstract

Policy regularization methods such as maximum entropy regularization are widely used in reinforcement learning to improve the robustness of a learned policy. In this paper, we show how this robustness arises from hedging against worst-case perturbations of the reward function, which are chosen from a limited set by an imagined adversary. Using convex duality, we characterize this robust set of adversarial reward perturbations under KL and alpha-divergence regularization, which includes Shannon and Tsallis entropy regularization as special cases. Importantly, generalization guarantees can be given within this robust set. We provide detailed discussion of the worst-case reward perturbations, and present intuitive empirical examples to illustrate this robustness and its relationship with generalization. Finally, we discuss how our analysis complements and extends previous results on adversarial reward robustness and path consistency optimality conditions.

Figures (12)

• Figure 1: Robust set$\mathcal{R}_{\pi}$ (red region) of perturbed reward functions to which a stochastic policy generalizes, in the sense of \ref{['eq:generalization']}. Red star indicates the worst-case perturbed reward $r^{\prime}_{\pi_*} = r - \Delta r_{\pi_*}$ (\ref{['prop:optimal_perturbations']}) chosen by the adversary. The robust set also characterizes the set of reward perturbations $\Delta r(a,s)$ that are feasible for the adversary, which differs based on the choice of regularization function, regularization strength $\beta$, and reference distribution $\pi_0$ (see \ref{['sec:visualizations_feasible']} and \ref{['fig:feasible_set_main']}). We show the robust set for the optimal single-step policy with value estimates $Q(a,s) = r(a,s)$ and kl divergence regularization to a uniform $\pi_0$, with $\beta = 1$. Our robust set is larger and has a qualitatively different shape compared to the robust set of derman2021twice (dotted lines, see \ref{['sec:related']}).
• Figure 1: Comparison to related work.
• Figure 2: Conjugate Function expressions for kl and $\alpha$-divergence regularization of either the policy $\pi(a|s)$ or occupancy $\mu(a,s)$. See \ref{['app:conj1']}-\ref{['app:conj4']} for derivations. The final column shows the optimizing argument in the definition of the conjugate function $\frac{1}{\beta}\Omega^{*}(\Delta r)$, for example $\mu_{\Delta r} \coloneqq \mathop{\mathrm{arg\,max}}\limits_{\mu} \langle \mu, \Delta r \rangle -\frac{1}{\beta}\Omega_{\mu_0}(\mu)$. Note that each conjugate expression for $\pi(a|s)$ regularization also contains an outer expectation over $\mu(s)$.
• Figure 2: Robust Set (red region) of perturbed reward functions to which a stochastic policy generalizes, in the sense that the policy is guaranteed to achieve an expected modified reward greater than or equal to the value of the regularized objective (\ref{['eq:generalization']}). The robust set characterizes the perturbed rewards which are feasible for the adversary. Red stars indicate the worst-case perturbed reward $r^{\prime}_{\pi_*} = r - \Delta r_{\pi_*}$ (\ref{['prop:optimal_perturbations']}). We show robust sets for the optimal $\pi_*(a|s)$ with fixed $Q(a,s)=r(a,s)$ values (blue star), where the optimal policy differs based on the regularization parameters $\alpha, \beta, \pi_0$ (see \ref{['eq:optimal_policy']}). The robust set is more restricted with decreasing regularization strength (increasing $\beta$), implying decreased generalization. Importantly, the slope of the robust set boundary can be linked to the action probabilities under the policy (see \ref{['sec:visualizations_feasible']}).
• Figure 3: Single-Step Reward Perturbations for kl regularization to uniform reference policy $\pi_0(a|s)$. $Q$-values in left columns are used for each $\beta$ in columns 2-4. We report the worst-case $-\Delta r_{\pi_*}(a,s)$ (\ref{['eq:optimal_perturbations']}), so negative values correspond to reward decreases. (a) Optimal policy ($Q_*(a,s) = r(a,s)$) using the environment reward, where the perturbed $r^{\prime}(a,s) = c \, \, \forall a$ reflects the indifference condition. (b) Suboptimal policy where indifference does not hold. In all cases, actions with high $Q(a,s)$ are robust to reward decreases.

Theorems & Definitions (24)

• Proposition 1
• Proposition 2
• Theorem 1: husain2021regularized
• Proposition 3
• proof
• Proposition 3
• proof
• Lemma 1: Flow Constraints Ensure Normalization
• proof
• Proposition 4: Optimal Policy in Regularized MDP