Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

Vanshaj Khattar, Ming Jin

TL;DR

Offline reinforcement learning suffers from limited data coverage and value overestimation. This paper introduces an offline framework called implicit actor-critic (iAC) that uses optimization-solution functions as a deterministic actor and a monotone reward-warped critic, with robustness guaranteed by exponentially decaying sensitivity (EDS) and enhanced by relative pessimism. Theoretical results provide a performance bound that quantifies the benefit of a longer horizon and the EDS parameter, and empirical evaluations on building energy management and supply chain tasks show iAC beating state-of-the-art offline RL baselines. The approach blends optimization-based control with pessimistic offline RL to deliver improved stability and performance in real-world datasets.

Abstract

Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods.

Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

TL;DR

Offline reinforcement learning suffers from limited data coverage and value overestimation. This paper introduces an offline framework called implicit actor-critic (iAC) that uses optimization-solution functions as a deterministic actor and a monotone reward-warped critic, with robustness guaranteed by exponentially decaying sensitivity (EDS) and enhanced by relative pessimism. Theoretical results provide a performance bound that quantifies the benefit of a longer horizon and the EDS parameter, and empirical evaluations on building energy management and supply chain tasks show iAC beating state-of-the-art offline RL baselines. The approach blends optimization-based control with pessimistic offline RL to deliver improved stability and performance in real-world datasets.

Abstract

Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods.
Paper Structure (23 sections, 10 theorems, 54 equations, 3 figures, 1 table)

This paper contains 23 sections, 10 theorems, 54 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Markov decision process and offline RL
  5. Optimization solution functions and EDS property
  6. Implicit Actor-Critic (iAC) framework
  7. Actor optimization model
  8. Critic for Q-function approximation
  9. Offline iAC with relative pessimism
  10. iAC algorithm and robustness of the actor
  11. Theoretical analysis for iAC
  12. Performance guarantee for iAC
  13. Experiments
  14. Building energy management---CityLearn challenge
  15. Supply chain management
  16. ...and 8 more sections

Key Result

Proposition 1

Let $e_{t,\tau}^{(K)}$ denote the prediction error at iteration $K$ at some time step $t$. If Assumptions asmptn:SmoothMDP-asmptn:LICQasmptn hold, and if there exist constants $H\geq 1$, $C_1>0$, and $\lambda \in (0,1)$ such that: $\sum_{\tau=0}^{T-t}\lambda^\tau e_{t,\tau}^{(K)}\leq \left(\frac{(1-

Figures (3)

  • Figure 1: iAC performance comparison, with total cost (left bar), peak demand (middle bar), and ramping cost (right bar). Lower is better. Average scores are reported across 10 runs with standard deviation as error bars.
  • Figure 2: Performance of different versions of iAC at different iterations.
  • Figure 3: Supply chain total profits and comparison with other offline RL baselines. Average scores and error bars are reported across 10 runs.

Theorems & Definitions (19)

  • Definition 1: Prediction error
  • Proposition 1: lin2022bounded
  • Lemma 1
  • Definition 2: xie2021bellman
  • Theorem IV.1
  • proof
  • Lemma 2: Monotonicity lemma
  • proof
  • Lemma 3
  • Lemma 4
  • ...and 9 more