Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch

Shangtong Zhang, Remi Tachet, Romain Laroche

TL;DR

This paper proves global optimality and finite-sample convergence for a three-timescale, off-policy actor-critic algorithm that does not apply density-ratio corrections for state-distribution mismatch. By combining a learned critic with stochastic updates and a decaying KL regularization term in the actor, and by developing a time-inhomogeneous stochastic approximation framework with uniform contraction, it shows that the critic tracks the value function and the actor converges to stationary points close to the optimum. The Soft Actor Critic extension demonstrates convergence under maximum-entropy regularization with decaying weight, and the results hold without requiring initial-state coverage or a unique optimal policy. Together, these contributions provide theoretical justification for practical off-policy learning without density-ratio corrections and introduce a robust analytical framework for time-inhomogeneous RL dynamics with multiple timescales.

Abstract

In this paper, we establish the global optimality and convergence rate of an off-policy actor critic algorithm in the tabular setting without using density ratio to correct the discrepancy between the state distribution of the behavior policy and that of the target policy. Our work goes beyond existing works on the optimality of policy gradient methods in that existing works use the exact policy gradient for updating the policy parameters while we use an approximate and stochastic update step. Our update step is not a gradient update because we do not use a density ratio to correct the state distribution, which aligns well with what practitioners do. Our update is approximate because we use a learned critic instead of the true value function. Our update is stochastic because at each step the update is done for only the current state action pair. Moreover, we remove several restrictive assumptions from existing works in our analysis. Central to our work is the finite sample analysis of a generic stochastic approximation algorithm with time-inhomogeneous update operators on time-inhomogeneous Markov chains, based on its uniform contraction properties.

Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch

TL;DR

This paper proves global optimality and finite-sample convergence for a three-timescale, off-policy actor-critic algorithm that does not apply density-ratio corrections for state-distribution mismatch. By combining a learned critic with stochastic updates and a decaying KL regularization term in the actor, and by developing a time-inhomogeneous stochastic approximation framework with uniform contraction, it shows that the critic tracks the value function and the actor converges to stationary points close to the optimum. The Soft Actor Critic extension demonstrates convergence under maximum-entropy regularization with decaying weight, and the results hold without requiring initial-state coverage or a unique optimal policy. Together, these contributions provide theoretical justification for practical off-policy learning without density-ratio corrections and introduce a robust analytical framework for time-inhomogeneous RL dynamics with multiple timescales.

Abstract

In this paper, we establish the global optimality and convergence rate of an off-policy actor critic algorithm in the tabular setting without using density ratio to correct the discrepancy between the state distribution of the behavior policy and that of the target policy. Our work goes beyond existing works on the optimality of policy gradient methods in that existing works use the exact policy gradient for updating the policy parameters while we use an approximate and stochastic update step. Our update step is not a gradient update because we do not use a density ratio to correct the state distribution, which aligns well with what practitioners do. Our update is approximate because we use a learned critic instead of the true value function. Our update is stochastic because at each step the update is done for only the current state action pair. Moreover, we remove several restrictive assumptions from existing works in our analysis. Central to our work is the finite sample analysis of a generic stochastic approximation algorithm with time-inhomogeneous update operators on time-inhomogeneous Markov chains, based on its uniform contraction properties.

Paper Structure

This paper contains 45 sections, 77 theorems, 457 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Stochastic Approximation with Time-Inhomogeneous Operators on Time-Inhomogeneous Markov Chains
  4. Off-Policy Actor Critic with Decaying KL Regularization
  5. Convergence of the Critic
  6. Convergence of the Actor
  7. Soft Actor Critic
  8. Related Work
  9. Experiments
  10. Conclusion
  11. Proofs of Section \ref{['sec sa']}
  12. Proof of Lemma \ref{['lem uniform mixing']}
  13. Proof of Theorem \ref{['thm sa convergence']}
  14. Proofs of Section \ref{['sec opac']}
  15. Proof of Lemma \ref{['lem uniform contraction opac']}
  16. ...and 30 more sections

Key Result

Lemma 0

(Uniform ergodicity implies uniform mixing) Let Assumption assu uniform ergodicity hold. Then, there exist constants $C_0 > 0$ and $\tau \in (0, 1)$, independent of $\theta$, such that for any $n > 0$,

Figures (3)

  • Figure 1: The chain domain from laroche2021dr with $\gamma = 0.99$.
  • Figure 2: Evaluation performance against training steps of Algorithm \ref{['alg opac']}
  • Figure 3: Evaluation performance against training steps of Algorithm \ref{['alg sac']}

Theorems & Definitions (77)

  • Lemma 0
  • Theorem 1
  • Lemma 1
  • Proposition 1
  • Theorem 2
  • Proposition 2
  • Theorem 3
  • Corollary 3
  • Lemma 3
  • Theorem 3
  • ...and 67 more