Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Universal Approximation Theorem of Deep Q-Networks

Qian Qi

TL;DR

The paper develops a continuous-time framework for Deep Q-Networks by linking deep reinforcement learning with stochastic control and Forward-Backward SDEs, enabling rigorous analysis of approximation and training dynamics. It proves a universal approximation theorem for residual-architecture DQNs, showing they can approximate the optimal Q-function $Q^*(t,s,a)$ on high-probability compact sets, with accuracy controlled by $\epsilon$ and local dynamics captured on $K_R$. It then establishes a convergence result for a continuous-time Q-learning scheme, asserting almost sure convergence of the parameter iterates to the optimal Q-function under ergodic sampling, Robbins-Monro steps, and gradient-based assumptions. By connecting the value function $V^*$ to a viscosity-solution formulation of the HJB equation and linking Q-learning to a Bellman operator contraction, the work provides a theoretical foundation for DQNs in continuous-time environments and informs architectural and algorithmic design for systems with physical dynamics or high-frequency data.

Abstract

We establish a continuous-time framework for analyzing Deep Q-Networks (DQNs) via stochastic control and Forward-Backward Stochastic Differential Equations (FBSDEs). Considering a continuous-time Markov Decision Process (MDP) driven by a square-integrable martingale, we analyze DQN approximation properties. We show that DQNs can approximate the optimal Q-function on compact sets with arbitrary accuracy and high probability, leveraging residual network approximation theorems and large deviation bounds for the state-action process. We then analyze the convergence of a general Q-learning algorithm for training DQNs in this setting, adapting stochastic approximation theorems. Our analysis emphasizes the interplay between DQN layer count, time discretization, and the role of viscosity solutions (primarily for the value function $V^*$) in addressing potential non-smoothness of the optimal Q-function. This work bridges deep reinforcement learning and stochastic control, offering insights into DQNs in continuous-time settings, relevant for applications with physical systems or high-frequency data.

Universal Approximation Theorem of Deep Q-Networks

TL;DR

The paper develops a continuous-time framework for Deep Q-Networks by linking deep reinforcement learning with stochastic control and Forward-Backward SDEs, enabling rigorous analysis of approximation and training dynamics. It proves a universal approximation theorem for residual-architecture DQNs, showing they can approximate the optimal Q-function on high-probability compact sets, with accuracy controlled by and local dynamics captured on . It then establishes a convergence result for a continuous-time Q-learning scheme, asserting almost sure convergence of the parameter iterates to the optimal Q-function under ergodic sampling, Robbins-Monro steps, and gradient-based assumptions. By connecting the value function to a viscosity-solution formulation of the HJB equation and linking Q-learning to a Bellman operator contraction, the work provides a theoretical foundation for DQNs in continuous-time environments and informs architectural and algorithmic design for systems with physical dynamics or high-frequency data.

Abstract

We establish a continuous-time framework for analyzing Deep Q-Networks (DQNs) via stochastic control and Forward-Backward Stochastic Differential Equations (FBSDEs). Considering a continuous-time Markov Decision Process (MDP) driven by a square-integrable martingale, we analyze DQN approximation properties. We show that DQNs can approximate the optimal Q-function on compact sets with arbitrary accuracy and high probability, leveraging residual network approximation theorems and large deviation bounds for the state-action process. We then analyze the convergence of a general Q-learning algorithm for training DQNs in this setting, adapting stochastic approximation theorems. Our analysis emphasizes the interplay between DQN layer count, time discretization, and the role of viscosity solutions (primarily for the value function ) in addressing potential non-smoothness of the optimal Q-function. This work bridges deep reinforcement learning and stochastic control, offering insights into DQNs in continuous-time settings, relevant for applications with physical systems or high-frequency data.
Paper Structure (22 sections, 6 theorems, 59 equations, 2 figures, 1 table)

This paper contains 22 sections, 6 theorems, 59 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notation and Setup
  4. Deep Q-Networks and Their Continuous-Time Representation
  5. Optimal Value and Action-Value Functions
  6. Q-Learning Algorithm and Assumptions
  7. Main Results
  8. Approximation Capability of DQNs
  9. Convergence of DQN Training
  10. Concluding Remarks
  11. Omitted Proofs
  12. Proof of Lemma \ref{['lem:resnet_suitability']}
  13. Proof of Lemma \ref{['lem:simultaneous_approximation']}
  14. Proof of Lemma \ref{['lem:universal_approximation_residual']}
  15. Proof of Lemma \ref{['lem:large_deviation_state_action']}
  16. ...and 7 more sections

Key Result

Lemma 2.6

Let Assumptions assump:state_action_spaces-assump:q_function hold, implying the optimal Q-function $Q^*$ is continuous on $[0,T] \times \mathcal{S} \times A$. Let $K_R = \{(t,s,a) \in [0,T] \times \mathcal{S} \times A : \|s\| \leq R_1, \|a\| \leq R_2\}$ be a compact subset relevant to the process dy

Figures (2)

  • Figure 1: Comparison of learning curves across different configurations. Top: Smoothed total episode rewards (window size = 20). Bottom: Average training loss per episode (log scale).
  • Figure 2: Comparison of learned policies. Shows the optimal action chosen by the agent for each state in $[-1, 1]$.

Theorems & Definitions (29)

  • Definition 2.4: Discrete-Time Deep Q-Network
  • Remark 2.5: Interpretation and Connection to SDE/FBSDE
  • Lemma 2.6: Suitability of ResNet Architecture for Approximating $Q^*$
  • proof
  • Lemma 2.7: Simultaneous Approximation
  • proof
  • Lemma 2.8: Universal Approximation by Residual Networks
  • proof
  • Remark 2.9: Expressiveness of the DQN
  • Lemma 2.10: Large Deviation Bound for State-Action Process
  • ...and 19 more