Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Identification of Latent Objectives in Stochastic Control

Yumiharu Nakano

TL;DR

Addresses identification of latent running costs $f$ and terminal costs $g$ that rationalize observed trajectory distributions for continuous-time controlled diffusions. It proposes a variational inverse problem based on the suboptimality gap $V(f,g)=\int_0^T\langle f(t,\cdot,\cdot), \mu_t\rangle dt + \langle g, \tilde{\mu}_T\rangle - J^*(f,g)$ and shows the optimal value $V^*$ equals the value of a generalized dynamic Schrödinger problem, connecting to stochastic optimal transport. Under mild conditions, a minimizer $(f^*,g^*)$ exists, and a one-dimensional linear-quadratic example demonstrates empirical parameter estimation. The work provides a principled framework for inverse inference in controlled diffusions and reveals a duality to Schrödinger bridges with relaxed marginals, broadening connections to stochastic OT.

Abstract

We propose a variational formulation of an inverse problem in continuous-time stochastic control, aimed at identifying control costs consistent with a given distribution over trajectories. The formulation is based on minimizing the suboptimality gap of observed behavior. We establish a connection between the inverse problem and a generalized dynamic Schrödinger problem, showing that their optimal values coincide. This result links inverse stochastic control with stochastic optimal transport, offering a new conceptual viewpoint on inverse inference in controlled diffusions.

On the Identification of Latent Objectives in Stochastic Control

TL;DR

Addresses identification of latent running costs and terminal costs that rationalize observed trajectory distributions for continuous-time controlled diffusions. It proposes a variational inverse problem based on the suboptimality gap and shows the optimal value equals the value of a generalized dynamic Schrödinger problem, connecting to stochastic optimal transport. Under mild conditions, a minimizer exists, and a one-dimensional linear-quadratic example demonstrates empirical parameter estimation. The work provides a principled framework for inverse inference in controlled diffusions and reveals a duality to Schrödinger bridges with relaxed marginals, broadening connections to stochastic OT.

Abstract

We propose a variational formulation of an inverse problem in continuous-time stochastic control, aimed at identifying control costs consistent with a given distribution over trajectories. The formulation is based on minimizing the suboptimality gap of observed behavior. We establish a connection between the inverse problem and a generalized dynamic Schrödinger problem, showing that their optimal values coincide. This result links inverse stochastic control with stochastic optimal transport, offering a new conceptual viewpoint on inverse inference in controlled diffusions.
Paper Structure (3 sections, 4 theorems, 53 equations)

This paper contains 3 sections, 4 theorems, 53 equations.

Table of Contents

  1. Introduction
  2. Existence
  3. Connection with stochastic optimal transport

Key Result

Proposition 2.1

Let $(A1)$ hold. Suppose that $\mu_t$ is the marginal law of $(X_t^{u^*}, u^*_t)$ for each $t\in [0,T]$, where $u^*=\{u^*_t\}_{0\le t\le T}\in\mathcal{U}$ is an optimal policy to the problem eq:1.1 for some $(f^*,g^*)\in\mathcal{D}$. Then, this $(f^*,g^*)$ is a minimizer for the inverse problem eq:1

Theorems & Definitions (10)

  • Proposition 2.1
  • Example 2.2
  • Proposition 2.3
  • proof
  • Example 2.4
  • Example 3.1
  • Theorem 3.2
  • proof
  • Theorem 3.3: mik-thi:2006
  • proof : Sketch of the proof