Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits

Sreejeet Maity, Aritra Mitra

TL;DR

This paper studies policy evaluation in RL under adversarial reward contamination with Markovian data. It first shows that vanilla TD(0) is vulnerable to Huber-type attacks, potentially converging to any limit, and then introduces Robust-TD, which uses robust mean estimation under Markovian data and a dynamic thresholding mechanism to provide finite-time guarantees that match TD(0) up to an additive $O(\epsilon)$ term. A matching minimax lower bound demonstrates the inevitability of this corruption-induced error, highlighting a fundamental trade-off between robustness and data corruption. The key technical contribution is a robust Median-of-Means analysis adapted to time-correlated data, enabling reliable value-function estimation despite adversarial perturbations. Together, these results advance robust reinforcement learning by characterizing both achievable guarantees and fundamental limits under adversarial reward contamination.

Abstract

One of the most basic problems in reinforcement learning (RL) is policy evaluation: estimating the long-term return, i.e., value function, corresponding to a given fixed policy. The celebrated Temporal Difference (TD) learning algorithm addresses this problem, and recent work has investigated finite-time convergence guarantees for this algorithm and variants thereof. However, these guarantees hinge on the reward observations being always generated from a well-behaved (e.g., sub-Gaussian) true reward distribution. Motivated by harsh, real-world environments where such an idealistic assumption may no longer hold, we revisit the policy evaluation problem from the perspective of adversarial robustness. In particular, we consider a Huber-contaminated reward model where an adversary can arbitrarily corrupt each reward sample with a small probability $ε$. Under this observation model, we first show that the adversary can cause the vanilla TD algorithm to converge to any arbitrary value function. We then develop a novel algorithm called Robust-TD and prove that its finite-time guarantees match that of vanilla TD with linear function approximation up to a small $O(ε)$ term that captures the effect of corruption. We complement this result with a minimax lower bound, revealing that such an additive corruption-induced term is unavoidable. To our knowledge, these results are the first of their kind in the context of adversarial robustness of stochastic approximation schemes driven by Markov noise. The key new technical tool that enables our results is an analysis of the Median-of-Means estimator with corrupted, time-correlated data that might be of independent interest to the literature on robust statistics.

Adversarially-Robust TD Learning with Markovian Data: Finite-Time Rates and Fundamental Limits

TL;DR

This paper studies policy evaluation in RL under adversarial reward contamination with Markovian data. It first shows that vanilla TD(0) is vulnerable to Huber-type attacks, potentially converging to any limit, and then introduces Robust-TD, which uses robust mean estimation under Markovian data and a dynamic thresholding mechanism to provide finite-time guarantees that match TD(0) up to an additive term. A matching minimax lower bound demonstrates the inevitability of this corruption-induced error, highlighting a fundamental trade-off between robustness and data corruption. The key technical contribution is a robust Median-of-Means analysis adapted to time-correlated data, enabling reliable value-function estimation despite adversarial perturbations. Together, these results advance robust reinforcement learning by characterizing both achievable guarantees and fundamental limits under adversarial reward contamination.

Abstract

One of the most basic problems in reinforcement learning (RL) is policy evaluation: estimating the long-term return, i.e., value function, corresponding to a given fixed policy. The celebrated Temporal Difference (TD) learning algorithm addresses this problem, and recent work has investigated finite-time convergence guarantees for this algorithm and variants thereof. However, these guarantees hinge on the reward observations being always generated from a well-behaved (e.g., sub-Gaussian) true reward distribution. Motivated by harsh, real-world environments where such an idealistic assumption may no longer hold, we revisit the policy evaluation problem from the perspective of adversarial robustness. In particular, we consider a Huber-contaminated reward model where an adversary can arbitrarily corrupt each reward sample with a small probability . Under this observation model, we first show that the adversary can cause the vanilla TD algorithm to converge to any arbitrary value function. We then develop a novel algorithm called Robust-TD and prove that its finite-time guarantees match that of vanilla TD with linear function approximation up to a small term that captures the effect of corruption. We complement this result with a minimax lower bound, revealing that such an additive corruption-induced term is unavoidable. To our knowledge, these results are the first of their kind in the context of adversarial robustness of stochastic approximation schemes driven by Markov noise. The key new technical tool that enables our results is an analysis of the Median-of-Means estimator with corrupted, time-correlated data that might be of independent interest to the literature on robust statistics.

Paper Structure

This paper contains 14 sections, 13 theorems, 106 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Model and Problem Formulation
  3. Motivation: Vulnerability of TD(0)
  4. Robust Markovian Mean Estimation
  5. Robust TD Learning Algorithm
  6. Main Results
  7. Conclusion and Future Work
  8. Additional Literature Review
  9. Useful Results
  10. Proof of Theorem \ref{['thm:vulnerability']}
  11. Performance Analysis for RUMEM: Proof of Theorem \ref{['theorem:RobustMean']}
  12. Main Convergence Analysis for Robust-TD: Proof of Theorem \ref{['thm:main_result']}
  13. Proof of Lower Bound in Theorem \ref{['thm:lowerbnd']}
  14. Simulation Study

Key Result

Theorem 1

(Vulnerability of TD(0)) Suppose Assumption ass:MC holds, and the step-size sequence $\{\alpha_t\}$ of TD(0) is chosen to satisfy $\sum_{t=0}^{\infty} \alpha_t = \infty$ and $\sum_{t=0}^{\infty} \alpha^2_t < \infty$. Then, under the Huber-contaminated reward model described above, the iterates of va Furthermore, for every point $w \in \mathbb{R}^K$, there exists a corresponding choice of corrupted

Figures (3)

  • Figure 1: Plot of mean-square error $E_t$ showing the effect of reward corruption on TD(0), with corruption probability $\epsilon =0.001$.
  • Figure 2: Plots of the mean-square error $E_t=\Vert \theta_T-\theta^* \Vert^2_2$ for TD(0) under the Huber-contaminated reward model with corruption probability $\epsilon=0.001$, and a simple biasing attack where the attack signal is $100/\epsilon.$ (Left) Constant step-size $\alpha=0.1$. (Right) Diminishing step-size $\alpha_t =1/t$.
  • Figure 3: Plots of the mean-square error $E_t=\Vert \theta_T-\theta^* \Vert^2_2$ for Robust-TD under the Huber-contaminated reward model with a biasing attack, where the attack signal is $100/\epsilon.$ (Left) Effect of varying the corruption probability $\epsilon$. (Right) Effect of varying the noise variance $\rho^2$.

Theorems & Definitions (19)

  • Theorem 1
  • Theorem 2
  • Definition 1
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • ...and 9 more