Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuity of Filters for Discrete-Time Control Problems Defined by Explicit Equations

Eugene A. Feinberg, Sayaka Ishizawa, Pavlo O. Kasyanov, David N. Kraemer

TL;DR

The paper addresses the problem of establishing weak continuity of the filter in discrete-time stochastic control with incomplete information, by relating stochastic-equation models to POMDP kernels through Aumann's lemma. It develops sufficient conditions on the transition $F$ and observation $G$ that guarantee the filter kernel $q$ is weakly continuous (and, under stronger conditions, continuous in total variation), enabling existence of optimal policies and convergence of value iterations for the reduced COMDP. A key technical advance is a criterion for total-variation continuity of kernels defined via stochastic equations, leveraging diffeomorphic image mappings and continuity of transformed neighborhoods. The results are illustrated on linear state-space models and inventory-control problems with noisy observations, and extended to POMDP1 formulations, providing practical criteria for ensuring optimal control under partial information in engineering, economics, and related fields.

Abstract

Discrete time control systems whose dynamics and observations are described by stochastic equations are common in engineering, operations research, health care, and economics. For example, stochastic filtering problems are usually defined via stochastic equations. These problems can be reduced to Markov decision processes (MDPs) whose states are posterior state distributions, and transition probabilities for such MDPs are sometimes called filters. This paper investigates sufficient conditions on transition and observation functions for the original problems to guarantee weak continuity of the filter. Under mild conditions on cost functions, weak continuity implies the existence of optimal policies minimizing the expected total costs, the validity of optimality equations, and convergence of value iterations to optimal values. This paper uses recent results on weak continuity of filters for partially observable MDPs defined by transition and observation probabilities. It develops a criterion of weak continuity of transition probabilities and a sufficient condition for continuity in total variation of transition probabilities. The results are illustrated with applications to filtering problems.

Continuity of Filters for Discrete-Time Control Problems Defined by Explicit Equations

TL;DR

The paper addresses the problem of establishing weak continuity of the filter in discrete-time stochastic control with incomplete information, by relating stochastic-equation models to POMDP kernels through Aumann's lemma. It develops sufficient conditions on the transition and observation that guarantee the filter kernel is weakly continuous (and, under stronger conditions, continuous in total variation), enabling existence of optimal policies and convergence of value iterations for the reduced COMDP. A key technical advance is a criterion for total-variation continuity of kernels defined via stochastic equations, leveraging diffeomorphic image mappings and continuity of transformed neighborhoods. The results are illustrated on linear state-space models and inventory-control problems with noisy observations, and extended to POMDP1 formulations, providing practical criteria for ensuring optimal control under partial information in engineering, economics, and related fields.

Abstract

Discrete time control systems whose dynamics and observations are described by stochastic equations are common in engineering, operations research, health care, and economics. For example, stochastic filtering problems are usually defined via stochastic equations. These problems can be reduced to Markov decision processes (MDPs) whose states are posterior state distributions, and transition probabilities for such MDPs are sometimes called filters. This paper investigates sufficient conditions on transition and observation functions for the original problems to guarantee weak continuity of the filter. Under mild conditions on cost functions, weak continuity implies the existence of optimal policies minimizing the expected total costs, the validity of optimality equations, and convergence of value iterations to optimal values. This paper uses recent results on weak continuity of filters for partially observable MDPs defined by transition and observation probabilities. It develops a criterion of weak continuity of transition probabilities and a sufficient condition for continuity in total variation of transition probabilities. The results are illustrated with applications to filtering problems.
Paper Structure (11 sections, 19 theorems, 57 equations)

This paper contains 11 sections, 19 theorems, 57 equations.

Table of Contents

  1. Introduction
  2. Notation and Background Definitions
  3. Continuity of Filter Kernels Generated by Stochastic Equations
  4. Semi-Uniform Feller Kernels and Their Properties
  5. Continuity of Stochastic Kernels Defined by Stochastic Equations
  6. Existence of Optimal Policies, Validity of Optimality Equations, and Convergence of Value Iterations for POMDPs
  7. Examples of Applications
  8. Linear state space models (LSSMs)
  9. Inventory models with noisy observations
  10. Another Form of Stochastic Equations
  11. Proofs of Corollary \ref{['cor:kernels:aumann_multidim']} and Theorems \ref{['thm:kernels:weak']}, \ref{['thm:kernels:total_variation']}, \ref{['thm:feller:filter_kernel_continuous']}, \ref{['thm:filters:main_result']}, and \ref{['thm:pomdp1s:results']}

Key Result

Theorem 3.3

Under Assumption ass:filters:assumption_1 or Assumption ass:filters:assumption_2, the stochastic kernel $q$ on $\mathbb{P}(\mathbb{X})$ given $\mathbb{P}(\mathbb{X}) \times \mathbb{A}$ defined in eq:filters:filter_kernel is weakly continuous.

Theorems & Definitions (48)

  • Definition 2.1: Continuity in distribution, total variation, and probability
  • Remark 2.2
  • Theorem 3.3: Weak continuity of the filter
  • proof
  • Definition 4.1: feinberg_markov_2022, feinberg_2023_semi
  • Theorem 4.2: feinberg_markov_2022
  • Theorem 4.3: feinberg_markov_2022
  • Theorem 4.4: Semi-uniform Feller of the stochastic kernel $P$
  • proof
  • Theorem 5.1: aumann_1964_mixed, gihman_1979_controlled
  • ...and 38 more