Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Approximation Theorem for Deep Q-Learning via FBSDE System

Qian Qi

TL;DR

This work establishes a universal approximation theorem tailored to a class of Deep Q-Networks whose layers are neural operators that mimic Bellman updates in a finite-horizon continuous-time MDP. By leveraging BSDE/FBSDE regularity, it proves uniform Lipschitz continuity and boundedness of the Bellman iterates $Q^{(k)}$ on the compact domain $K_Q$ and shows that a deep residual network can approximate the Bellman residual $\mathcal{B}Q-Q$ with controlled error, linking network depth to iterative value refinement. The key contributions include a problem-aware UAT for neural-operator DQNs, a contraction-based analysis of the Bellman operator, and a framework for analyzing error propagation across layers, along with discussion on quantifying rates and strategies to mitigate the curse of dimensionality. This approach provides a principled, dynamic-systems perspective on how depth and regularity govern the approximation of the optimal Q-function, with implications for designing scalable, structure-aware reinforcement learning architectures in high dimensions.

Abstract

The approximation capabilities of Deep Q-Networks (DQNs) are commonly justified by general Universal Approximation Theorems (UATs) that do not leverage the intrinsic structural properties of the optimal Q-function, the solution to a Bellman equation. This paper establishes a UAT for a class of DQNs whose architecture is designed to emulate the iterative refinement process inherent in Bellman updates. A central element of our analysis is the propagation of regularity: while the transformation induced by a single Bellman operator application exhibits regularity, for which Backward Stochastic Differential Equations (BSDEs) theory provides analytical tools, the uniform regularity of the entire sequence of value iteration iterates--specifically, their uniform Lipschitz continuity on compact domains under standard Lipschitz assumptions on the problem data--is derived from finite-horizon dynamic programming principles. We demonstrate that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator. The resulting approximation theorem is thus intrinsically linked to the control problem's structure, offering a proof technique wherein network depth directly corresponds to iterations of value function refinement, accompanied by controlled error propagation. This perspective reveals a dynamic systems view of the network's operation on a space of value functions.

Universal Approximation Theorem for Deep Q-Learning via FBSDE System

TL;DR

This work establishes a universal approximation theorem tailored to a class of Deep Q-Networks whose layers are neural operators that mimic Bellman updates in a finite-horizon continuous-time MDP. By leveraging BSDE/FBSDE regularity, it proves uniform Lipschitz continuity and boundedness of the Bellman iterates on the compact domain and shows that a deep residual network can approximate the Bellman residual with controlled error, linking network depth to iterative value refinement. The key contributions include a problem-aware UAT for neural-operator DQNs, a contraction-based analysis of the Bellman operator, and a framework for analyzing error propagation across layers, along with discussion on quantifying rates and strategies to mitigate the curse of dimensionality. This approach provides a principled, dynamic-systems perspective on how depth and regularity govern the approximation of the optimal Q-function, with implications for designing scalable, structure-aware reinforcement learning architectures in high dimensions.

Abstract

The approximation capabilities of Deep Q-Networks (DQNs) are commonly justified by general Universal Approximation Theorems (UATs) that do not leverage the intrinsic structural properties of the optimal Q-function, the solution to a Bellman equation. This paper establishes a UAT for a class of DQNs whose architecture is designed to emulate the iterative refinement process inherent in Bellman updates. A central element of our analysis is the propagation of regularity: while the transformation induced by a single Bellman operator application exhibits regularity, for which Backward Stochastic Differential Equations (BSDEs) theory provides analytical tools, the uniform regularity of the entire sequence of value iteration iterates--specifically, their uniform Lipschitz continuity on compact domains under standard Lipschitz assumptions on the problem data--is derived from finite-horizon dynamic programming principles. We demonstrate that layers of a deep residual network, conceived as neural operators acting on function spaces, can approximate the action of the Bellman operator. The resulting approximation theorem is thus intrinsically linked to the control problem's structure, offering a proof technique wherein network depth directly corresponds to iterations of value function refinement, accompanied by controlled error propagation. This perspective reveals a dynamic systems view of the network's operation on a space of value functions.
Paper Structure (25 sections, 4 theorems, 33 equations, 1 figure)

This paper contains 25 sections, 4 theorems, 33 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Continuous-Time Markov Decision Process
  4. Forward-Backward Stochastic Differential Equations (FBSDEs)
  5. Deep Q-Network Architecture (Operator-based ResNet)
  6. Optimal Q-function via an Iterative Scheme with BSDE-Inspired Regularity
  7. Universal Approximation via FBSDE-Inspired Network Construction
  8. Quantitative Approximation Rates and Curse of Dimensionality
  9. Quantitative Approximation Rates
  10. Assumptions on Higher-Order Smoothness
  11. Bounds on Discretization and Interpolation Error
  12. Bounds on Neural Network ($\mathcal{N}_{\theta_l}$) Approximation Error
  13. Overall Rate and Dependence on Parameters
  14. Implications for Mitigating the Curse of Dimensionality
  15. Leveraging Smoothness for Advanced Discretization
  16. ...and 10 more sections

Key Result

Lemma 3.1

Let Assumption assump:mdp_coeffs hold. Let $K_Q$ be compact. Assume $\lambda > 0$.

Figures (1)

  • Figure 1: Illustration of Bellman iterates $Q^{(k)}(s,a)$ for $k=0, 1, 2$. The state space is $s \in [0,1]$. Actions $a_L$ (move left) and $a_R$ (move right) are shown. $Q^{(0)}$ is zero. $Q^{(1)}$ is identical for both actions. $Q^{(2)}$ shows distinct values for $a_L$ and $a_R$, reflecting the one-step lookahead with $V^{(1)}$. All iterates are bounded and visually Lipschitz continuous.

Theorems & Definitions (13)

  • Remark 2.2: On Assumption \ref{['assump:mdp_coeffs']}
  • Lemma 3.1: Properties of $\mathcal{B}$ and Iterates $Q^{(k)}$
  • proof
  • Lemma 3.2: Compactness of Iterates
  • proof
  • Remark 4.2: On the determination of $L_F^*$
  • Lemma 4.3: UAT for Neural Operators Approximating $\mathcal{J}$
  • proof
  • Theorem 4.4: UAT for DQNs via Iterative Refinement and Regularity Propagation
  • proof
  • ...and 3 more