Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quadratic Truncated Random Return in Distributional LQR: Positive Definiteness, Density, and Log-Concavity

Ruyi Teng, Dan Wang, Wei Chen, Yulong Gao

TL;DR

This work analyzes the truncated random return in distributional LQR by recasting it as a quadratic form $S(Z)=Z^ op H_N Z+2L_N^ op Z+c$ in the disturbance vector $Z$ and establishing a positive definiteness result for $H_N$ under a sufficient condition. It proves that, with zero-mean Gaussian disturbances, the truncated return $G_N^{K}(x)$ follows a positively weighted non-central chi-square distribution, and that the CDF of $G_N^{K}(x)$ is log-concave if disturbance PDFs are log-concave, via Prékopa's theorem. The paper also provides eigenvalue bounds for $H_N$, a scalar-system positivity guarantee, and a data-center cooling example to validate the theory. These results yield tractable density expressions and robust distributional properties, enabling risk-aware analysis and sampling for truncated distributional LQR. Overall, the work advances understanding of the stochastic structure of the distributional LQR's finite-horizon approximation and its implications for risk-sensitive control.

Abstract

Distributional linear quadratic regulator (LQR) is a new framework that integrates the distributional reinforcement learning and classical LQR, which offers a new way to study the random return instead of the expected cost. Unlike iterative approximation using dynamic programming in the DRL, a closed-form expression for the random return can be exactly characterized in the distributional LQR, which is defined over infinitely many random variables. In recent work [1, 2], it has been shown that this random return can be well approximated by a finite number of random variables, which we call truncated random return. In this paper, we study the truncated random return in the distributional LQR. We show that the truncated random return can be naturally expressed in the quadratic form. We develop a sufficient condition for the positive definiteness of the block symmetric matrix in the quadratic form and provide the lower and upper bounds on the eigenvalues of this matrix. We further show that in this case, the truncated random return follows a positively weighted non-central chi-square distribution if the random disturbances admits Gaussian, and its cumulative distribution function is log-concave if the probability density function of the random disturbances is log-concave.

Quadratic Truncated Random Return in Distributional LQR: Positive Definiteness, Density, and Log-Concavity

TL;DR

This work analyzes the truncated random return in distributional LQR by recasting it as a quadratic form in the disturbance vector and establishing a positive definiteness result for under a sufficient condition. It proves that, with zero-mean Gaussian disturbances, the truncated return follows a positively weighted non-central chi-square distribution, and that the CDF of is log-concave if disturbance PDFs are log-concave, via Prékopa's theorem. The paper also provides eigenvalue bounds for , a scalar-system positivity guarantee, and a data-center cooling example to validate the theory. These results yield tractable density expressions and robust distributional properties, enabling risk-aware analysis and sampling for truncated distributional LQR. Overall, the work advances understanding of the stochastic structure of the distributional LQR's finite-horizon approximation and its implications for risk-sensitive control.

Abstract

Distributional linear quadratic regulator (LQR) is a new framework that integrates the distributional reinforcement learning and classical LQR, which offers a new way to study the random return instead of the expected cost. Unlike iterative approximation using dynamic programming in the DRL, a closed-form expression for the random return can be exactly characterized in the distributional LQR, which is defined over infinitely many random variables. In recent work [1, 2], it has been shown that this random return can be well approximated by a finite number of random variables, which we call truncated random return. In this paper, we study the truncated random return in the distributional LQR. We show that the truncated random return can be naturally expressed in the quadratic form. We develop a sufficient condition for the positive definiteness of the block symmetric matrix in the quadratic form and provide the lower and upper bounds on the eigenvalues of this matrix. We further show that in this case, the truncated random return follows a positively weighted non-central chi-square distribution if the random disturbances admits Gaussian, and its cumulative distribution function is log-concave if the probability density function of the random disturbances is log-concave.
Paper Structure (10 sections, 9 theorems, 37 equations, 2 figures, 1 table)

This paper contains 10 sections, 9 theorems, 37 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Statement
  3. Notation
  4. Canonical discounted LQR
  5. Distributional LQR
  6. Main Results
  7. Positive definiteness
  8. Probability density function
  9. Log-concavity of CDF
  10. Conclusion

Key Result

Lemma II.1

feingold1962block Let $H=[H_{ij}]$ be a block matrix where $H_{ij}\in \mathbb{R}^{n\times n}$, $i,j=1,\ldots, N$. For each eigenvalue $\lambda$ of $H$, there exists at least one $1\leq i\leq N$ such that

Figures (2)

  • Figure 1: Probability density values of $G_N^K(x)$ where the random disturbance is zero-mean Gaussian.
  • Figure 2: Second derivative of $\operatorname{log}(F_N(g))$ under five different random disturbances whose PDFs are log-concave

Theorems & Definitions (24)

  • Lemma II.1
  • Definition II.1
  • Proposition III.1
  • proof
  • Remark III.1
  • Lemma III.1
  • proof
  • Theorem III.1
  • proof
  • Proposition III.2
  • ...and 14 more