Reinforcement Learning from Partial Observation: Linear Function Approximation with Provable Sample Efficiency

TL;DR

This work addresses reinforcement learning in partially observed MDPs with infinite observation and state spaces by introducing a linear-structure POMDP and the OP-TENET algorithm. OP-TENET combines a finite-memory Bellman operator, adversarial integral equation estimation via minimax optimization with RKHS-based projection, and optimistic exploration to achieve provable sample efficiency. The authors prove that OP-TENET attains an $ε$-optimal policy in $O(1/ε^2)$ episodes, with a sample complexity that scales polynomially in the intrinsic dimension of the linear structure and is independent of the actual sizes of the observation and state spaces. This provides a theoretical foundation for learning under partial observability with function approximation and offers a concrete algorithmic blueprint for efficient exploration and model estimation in POMDPs.

Abstract

We study reinforcement learning for partially observed Markov decision processes (POMDPs) with infinite observation and state spaces, which remains less investigated theoretically. To this end, we make the first attempt at bridging partial observability and function approximation for a class of POMDPs with a linear structure. In detail, we propose a reinforcement learning algorithm (Optimistic Exploration via Adversarial Integral Equation or OP-TENET) that attains an $ε$-optimal policy within $O(1/ε^2)$ episodes. In particular, the sample complexity scales polynomially in the intrinsic dimension of the linear structure and is independent of the size of the observation and state spaces. The sample efficiency of OP-TENET is enabled by a sequence of ingredients: (i) a Bellman operator with finite memory, which represents the value function in a recursive manner, (ii) the identification and estimation of such an operator via an adversarial integral equation, which features a smoothed discriminator tailored to the linear structure, and (iii) the exploration of the observation and state spaces via optimism, which is based on quantifying the uncertainty in the adversarial integral equation.

Figures (2)

• Figure 1: Directed acyclic graph of a POMDP. Here, we denote by $\bm{\tau}_h=(\bm{o}_1,\ldots,\bm{o}_h)$ the observation history and ${\overline{\bm{\tau}}}_h=(\bm{o}_1,\bm{a}_1,\ldots,\bm{o}_{h-1},\bm{a}_{h-1},\bm{o}_h)$ the full history. See Section \ref{['prel1']} for more details.
• Figure 2: Illustration of the variables in the definition of ${\mathbb{B}}^{\theta,\pi}_h$ in \ref{['bdef']} and \ref{['b-func-def']}. In detail, $\widetilde{\bm{s}}_h$ is an independent replicate of $\bm{s}_{h}$, that is, they are independent and identically distributed conditioning on $\bm{s}_{h-1}$ and $\bm{a}_{h-1}$. Note that $\widetilde{\bm{s}}_h$ is constructed for ease of presentation, and does not exist in practice. Then, the action $\widetilde{\bm{a}}_{h}$, state $\widetilde{\bm{s}}_{h+1}$, and observations $\widetilde{\bm{o}}_{h}, \widetilde{\bm{o}}_{h+1}$ are similarly defined. In other words, their distribution conditioning on $\widetilde{\bm{s}}_h$ and ${\overline{\bm{\tau}}}_{h-1}$ mirrors the distribution of the action $\bm{a}_{h}$, state $\bm{s}_{h+1}$, and observations $\bm{o}_{h}, \bm{o}_{h+1}$ conditioning on $\bm{s}_h$ and ${\overline{\bm{\tau}}}_{h-1}$. For notational simplicity, we define the tail-mirrored full history ${\overline{\bm{\tau}}}_{h}^\dagger=({\overline{\bm{\tau}}}_{h-1}, \bm{a}_{h-1}, \widetilde{\bm{o}}_h)$ and tail-mirrored observation history $\bm{\tau}_{h}^\dagger=(\bm{\tau}_{h-1}, \widetilde{\bm{o}}_h)$.

Theorems & Definitions (22)

• Lemma 3.1: Operators Equivalence
• Corollary 3.2
• Lemma 3.3
• Theorem 4.1
• Lemma 4.2: Value Decomposition
• Lemma 4.3: Statistical Guarantee
• Lemma 4.4: Telescope of Error
• Definition A.1: Linear Kernel POMDPs
• Lemma A.2
• Lemma A.3