Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent

Nicole Bäuerle, Athanasios Vasileiadis

TL;DR

The paper defines Distributional MDPs of the third kind where the goal is to steer the distribution of the terminal cumulative reward toward a target law. It introduces a model-free policy-gradient method with randomized Markov policies parameterized by neural networks on an augmented state space, optimizing a characteristic-function loss to match the target distribution. A lifted MDP formulation, rigorous convergence guarantees to stationary points via stochastic approximation, and a suite of applications (linear-quadratic, wealth targeting, and compact-support distributions) demonstrate the method's flexibility and highlight non-uniqueness phenomena in distributional controls. The approach provides a principled, data-driven way to shape full reward distributions and connects classical MDPs to distributional objectives through augmentation and CF-based objectives. Overall, the work advances distributional control by coupling neural-parametrized randomized policies with CF-based objectives and proving convergence in a model-free setting, with practical impact for risk-aware decision-making and distributional policy design.

Abstract

The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law, rather than optimizing an expected value or a risk functional. To solve the resulting distributional control problem in a model-free setting, we propose a policy-gradient algorithm based on neural-network parameterizations of randomized Markov policies, defined on an augmented state space and a sample-based evaluation of the characteristic-function loss. Under mild regularity and growth assumptions, we prove convergence of the algorithm to stationary points using stochastic approximation techniques. Several numerical experiments illustrate the ability of the method to match complex target distributions, recover classical optimal policies when they exist, and reveal intrinsic non-uniqueness phenomena specific to distributional control.

Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent

TL;DR

The paper defines Distributional MDPs of the third kind where the goal is to steer the distribution of the terminal cumulative reward toward a target law. It introduces a model-free policy-gradient method with randomized Markov policies parameterized by neural networks on an augmented state space, optimizing a characteristic-function loss to match the target distribution. A lifted MDP formulation, rigorous convergence guarantees to stationary points via stochastic approximation, and a suite of applications (linear-quadratic, wealth targeting, and compact-support distributions) demonstrate the method's flexibility and highlight non-uniqueness phenomena in distributional controls. The approach provides a principled, data-driven way to shape full reward distributions and connects classical MDPs to distributional objectives through augmentation and CF-based objectives. Overall, the work advances distributional control by coupling neural-parametrized randomized policies with CF-based objectives and proving convergence in a model-free setting, with practical impact for risk-aware decision-making and distributional policy design.

Abstract

The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law, rather than optimizing an expected value or a risk functional. To solve the resulting distributional control problem in a model-free setting, we propose a policy-gradient algorithm based on neural-network parameterizations of randomized Markov policies, defined on an augmented state space and a sample-based evaluation of the characteristic-function loss. Under mild regularity and growth assumptions, we prove convergence of the algorithm to stationary points using stochastic approximation techniques. Several numerical experiments illustrate the ability of the method to match complex target distributions, recover classical optimal policies when they exist, and reveal intrinsic non-uniqueness phenomena specific to distributional control.
Paper Structure (38 sections, 11 theorems, 136 equations, 12 figures, 1 algorithm)

This paper contains 38 sections, 11 theorems, 136 equations, 12 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Extension to Continuous Non Compact Spaces
  4. Choosing the Target
  5. Literature Review
  6. Summary and Organisation of the Article
  7. The Problem and Assumptions
  8. Problem Formulation
  9. Lifted Distributional MDP Formulation
  10. Controls
  11. Assumptions and Some Implications
  12. Assumptions on the neural net data
  13. Assumptions on the growth of the state process
  14. Further Lipschitz-Continuity Assumptions on the Data
  15. Bounds on the evolution of the process
  16. ...and 23 more sections

Key Result

Theorem 6

Under Assumptions 1-5 the mapping $\theta \mapsto \nabla_\theta \mathcal{L}_{L}(\theta)$ is globally Lipschitz continuous on $\Theta$ with a constant $\mathbf{L}>0$, i.e.

Figures (12)

  • Figure 1: Distribution of Rewards
  • Figure 2: Two different optimal action distributions
  • Figure 3: Distribution of terminal wealth
  • Figure 4: Optimal actions distributions for Different wealth states
  • Figure 5: Distribution of terminal wealth
  • ...and 7 more figures

Theorems & Definitions (25)

  • Remark 2
  • Remark 3: Error induced by conditioning on $G$
  • Remark 4
  • Remark 5
  • Theorem 6
  • Lemma 7
  • Theorem 8: Convergence to a stationary point
  • proof
  • Remark 9
  • Remark 10: about the fit of the action distributions
  • ...and 15 more