Finite Memory Belief Approximation for Optimal Control in Partially Observable Markov Decision Processes
Mintae Kim
TL;DR
This work addresses the challenge of finite memory belief approximation in partially observable stochastic optimal control by treating truncated IO histories as finite-memory belief approximations and measuring their discrepancy from the true belief with the Wasserstein-2 distance. It develops a policy-conditional, fixed-policy comparison framework that relates belief mismatch along a closed-loop trajectory to the degradation in performance, yielding a bound $|J(\pi)-\hat{J}_H(\pi)| \le \frac{C_\pi}{1-\gamma}\, \varepsilon_H(\pi)$ and, under exponential forgetting, an exponential decay of the optimality gap with memory length. The paper demonstrates an exponential decay of the belief mismatch in $H$ for stabilizing policies and provides a concrete LQG specialization with closed-form belief and cost analyses, validated by numerical experiments. Overall, the results offer a metric-aware, quantitative account of what finite memory belief approximation can achieve in POSOC and highlight fundamental limits, such as the necessity of preserving input history and potential exponential memory requirements in general PO settings.
Abstract
We study finite memory belief approximation for partially observable (PO) stochastic optimal control (SOC) problems. While belief states are sufficient for SOC in partially observable Markov decision processes (POMDPs), they are generally infinite-dimensional and impractical. We interpret truncated input-output (IO) histories as inducing a belief approximation and develop a metric-based theory that directly relates information loss to control performance. Using the Wasserstein metric, we derive policy-conditional performance bounds that quantify value degradation induced by finite memory along typical closed-loop trajectories. Our analysis proceeds via a fixed-policy comparison: we evaluate two cost functionals under the same closed-loop execution and isolate the effect of replacing the true belief by its finite memory approximation inside the belief-level cost. For linear quadratic Gaussian (LQG) systems, we provide closed-form belief mismatch evaluation and empirically validate the predicted mechanism, demonstrating that belief mismatch decays approximately exponentially with memory length and that the induced performance mismatch scales accordingly. Together, these results provide a metric-aware characterization of what finite memory belief approximation can and cannot achieve in PO settings.