Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off

TL;DR

This work tackles the problem of data-efficient policy evaluation for continuous-time systems by analyzing Monte-Carlo evaluation in stochastic LQR/Langevin settings. The authors derive a closed-form mean-squared error surface that decomposes into approximation (discretization) and estimation (variance) components, showing that finer time steps reduce model error but increase variance under a fixed data budget, yielding an optimal sampling step $h^*$ that scales with the budget $B$. They extend the analysis from a one-dimensional Langevin process to multi-dimensional vector cases and both finite- and infinite-horizon objectives, including discounted settings, establishing scaling laws $h^*(B)\sim B^{-1/3}$ (finite horizon) and $h^*(B)\sim B^{-1/5}$ (infinite horizon). Numerical experiments on linear and nonlinear dynamical systems, including MuJoCo benchmarks, validate the theory and demonstrate practical guidelines for choosing sampling frequencies to improve data efficiency. The results have direct implications for RL practice, suggesting that practitioners should adapt temporal resolution to available data rather than rely on a fixed step-size. Extensions to policy optimization, adaptive sampling, and broader noise models are promising directions for future work.

Abstract

A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.

Figures (5)

• Figure 1: Mean-squared error trade-off in linear quadratic systems of different dimension $n$. The first two plots show the dependence of the optimal step-size on the data budget $B$ and drift coefficient $a$, respectively. A{1,2,3,4,5} in the last two plots are random matrices and the two sets are not equal.
• Figure 2: MSE of Monte-Carlo policy evaluation in nonlinear systems. The line and shaded region denote the sample mean and its standard error of $(\hat{V}_M(h) - V)^2$, from 30 random runs. $T$ is the horizon in physical time (seconds). $B_0$ denotes the environment-dependent base sample budget, chosen such that it gives a full episode for the smallest $h$ (see \ref{['app:experiment']}). The optimal step-size generally decreases as the data budget increases (with 'InvertedDoublePendulum-v2' being the only exception).
• Figure 3: Empirical $h^*$ in nonlinear experiments (solid) compared with analysis in \ref{['cor:optimal-step-size-vector']} (dashed): $h^* = c_FB^{-1/3}$, $c_F$ is estimated from data by least squares.
• Figure 4: Mean-squared error trade-off in LQR with scaled identity matrices A. The plots show the dependence of the optimal step-size on the eigenvalues of the linear systems in both finite and infinite horizon settings. The same trend of the scalar case w.r.t. the parameter $a$ can be observed here.
• Figure 5: Comparison between the empirical (solid) and analytical MSEs (dashed) in one-dimensional Langevin systems.

Theorems & Definitions (18)

• Theorem 3.1: Finite-horizon, undiscounted MSE
• Corollary 3.2: MSE for marginally stable system
• Corollary 3.3: Approximate MSE
• Theorem 3.4: Mean-squared error - vector case
• Corollary 3.5: Optimal step size - vector case
• Theorem 3.6: Infinite-horizon, discounted MSE
• Corollary 3.7
• Lemma A.1
• proof
• proof : Proof of \ref{['cor: MSE A=0 case']}