Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Episodic Convex Reinforcement Learning

Bianca Marin Moreno, Khaled Eldowa, Pierre Gaillard, Margaux Brégère, Nadia Oudjane

TL;DR

The paper tackles online CURL in episodic finite-horizon MDPs where the loss at each episode is a convex function of the policy-induced state-action occupancy. It introduces Bonus O-MD-CURL, an online mirror-descent algorithm that incorporates carefully designed exploration bonuses to cope with unknown dynamics, achieving sublinear regret in the full-information setting and extending to bandit feedback settings via gradient estimation. For full-information CURL with unknown p, the authors prove a near-optimal regret of $ ilde{O}(L N^3 |𝒳|^{3/2} \,√{|𝒜| T})$, using a varying-constraint MD framework and a closed-form update. They also develop two bandit-CURL approaches: (i) an entropic-regularization method providing a $ ilde{O}$(√{L(L+1)/ε} |𝒳|^{5/4} |𝒜|^{5/4} N^3 T^{3/4}) + … bound under a state-distribution assumption, and (ii) a self-concordant, barrier-based method for known MDPs achieving $ ilde{O}(√{L} N^{7/4} (|𝒳||𝒜| T)^{3/4})$. The work also includes RL-specific bandit results and empirical validation showing exploration bonuses materially improve learning in multi-objective and constrained tasks. Overall, the paper provides a theoretically grounded, practical framework for online CURL under unknown dynamics and bandit feedback, bridging convex optimization techniques with reinforcement learning in a principled way.

Abstract

We study online learning in episodic finite-horizon Markov decision processes (MDPs) with convex objective functions, known as the concave utility reinforcement learning (CURL) problem. This setting generalizes RL from linear to convex losses on the state-action distribution induced by the agent's policy. The non-linearity of CURL invalidates classical Bellman equations and requires new algorithmic approaches. We introduce the first algorithm achieving near-optimal regret bounds for online CURL without any prior knowledge on the transition function. To achieve this, we use an online mirror descent algorithm with varying constraint sets and a carefully designed exploration bonus. We then address for the first time a bandit version of CURL, where the only feedback is the value of the objective function on the state-action distribution induced by the agent's policy. We achieve a sub-linear regret bound for this more challenging problem by adapting techniques from bandit convex optimization to the MDP setting.

Online Episodic Convex Reinforcement Learning

TL;DR

The paper tackles online CURL in episodic finite-horizon MDPs where the loss at each episode is a convex function of the policy-induced state-action occupancy. It introduces Bonus O-MD-CURL, an online mirror-descent algorithm that incorporates carefully designed exploration bonuses to cope with unknown dynamics, achieving sublinear regret in the full-information setting and extending to bandit feedback settings via gradient estimation. For full-information CURL with unknown p, the authors prove a near-optimal regret of , using a varying-constraint MD framework and a closed-form update. They also develop two bandit-CURL approaches: (i) an entropic-regularization method providing a (√{L(L+1)/ε} |𝒳|^{5/4} |𝒜|^{5/4} N^3 T^{3/4}) + … bound under a state-distribution assumption, and (ii) a self-concordant, barrier-based method for known MDPs achieving . The work also includes RL-specific bandit results and empirical validation showing exploration bonuses materially improve learning in multi-objective and constrained tasks. Overall, the paper provides a theoretically grounded, practical framework for online CURL under unknown dynamics and bandit feedback, bridging convex optimization techniques with reinforcement learning in a principled way.

Abstract

We study online learning in episodic finite-horizon Markov decision processes (MDPs) with convex objective functions, known as the concave utility reinforcement learning (CURL) problem. This setting generalizes RL from linear to convex losses on the state-action distribution induced by the agent's policy. The non-linearity of CURL invalidates classical Bellman equations and requires new algorithmic approaches. We introduce the first algorithm achieving near-optimal regret bounds for online CURL without any prior knowledge on the transition function. To achieve this, we use an online mirror descent algorithm with varying constraint sets and a carefully designed exploration bonus. We then address for the first time a bandit version of CURL, where the only feedback is the value of the objective function on the state-action distribution induced by the agent's policy. We achieve a sub-linear regret bound for this more challenging problem by adapting techniques from bandit convex optimization to the MDP setting.
Paper Structure (68 sections, 37 theorems, 264 equations, 4 figures, 1 table, 4 algorithms)

This paper contains 68 sections, 37 theorems, 264 equations, 4 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Setting
  5. Preliminary Results
  6. Exploration Bonus in CURL
  7. Limitations of previous approaches
  8. CURL with full-information feedback
  9. Bandit Feedback
  10. Bandit feedback with bonus in RL
  11. CURL with bandit feedback
  12. Entropic Regularization Method
  13. Self-concordant regularization Method
  14. Experiments
  15. Auxiliary results
  16. ...and 53 more sections

Key Result

Lemma 2.1

Let $(q^t)_{t \in [T]}$ be a sequence of probability transition kernels and $(z^t)_{t \in [T]}$ a sequence of vectors in $\mathbb{R}^{N \times |\mathcal{X}| \times |\mathcal{A}|}$, such that $\max_{t\in [T] }\|z^t\|_{1,\infty} \leq \zeta$. Initialize $\pi^1_n(a|x) := 1/|\mathcal{A}|$. For $t \in [T] Then, there is a $\tau > 0$ such that, for any sequence $(\nu^t)_{t \in [T]}$, with $\nu^t := \nu^{

Figures (4)

  • Figure 1: [left] Initial agent distribution; [middle] The three targets from multi-objectives; [right] The constrained MDP (reward in yellow, constraints in blue).
  • Figure 2: Multi-objective: distribution at $N = 40$ after $50$ iters. for Bonus O-MD-CURL [up,left], Greedy MD-CURL [up,right]; log-loss [down,left] and regret [down,right] for $10^3$ iters.
  • Figure 3: Constrained MDP after $10^3$ iters.: sum distributions over all time steps $n \in [40]$ at [up,left]; distribution at the last time step $N = 40$ for Bonus O-MD-CURL [up,center], and Greedy MD-CURL [up,right]; the log-loss [down, left] and regret [down,right].
  • Figure 4: This figure provides a graphical comparison between the sampling approach used in \ref{['alg:md-curl-bandits-unknown-mdp']}, represented on the left, and that used in \ref{['rl:alg:md-curl-bandits-lb']}, represented on the right. The simplified domain here is $\{x \in [0,1]^{2} \colon \|x\|_1 \leq 1\}$. Both approaches are illustrated at three points: $a$, $b$, and $c$. In the first approach, with some $\delta \in (0,1)$ and $\bar{\delta} \coloneqq 1-\delta$, we sample from a circle of radius ${\delta}/(2+\sqrt{2})$ centered at a convex combination between the point of interest and $o \coloneqq \bigl({1}/(2+\sqrt{2}),{1}/(2+\sqrt{2})\bigr)$. In the second approach, we consider the barrier $-\log(1-x_1-x_2) -\sum_{i=1,2}\log(x_i)$ and sample from the Dikin ellipsoid (of a certain common radius) induced by this function at each point.

Theorems & Definitions (67)

  • Lemma 2.1
  • Lemma 2.2: Lem. 17 of UCRL-2
  • Proposition 3.1
  • Theorem 3.2
  • Theorem 4.1
  • Theorem 4.3
  • Theorem 4.5
  • Lemma 1.1
  • proof
  • Lemma 1.2
  • ...and 57 more