Robust Adversarial Reinforcement Learning in Stochastic Games via Sequence Modeling

TL;DR

This work addresses adversarial robustness of sequence-modeling RL, specifically the Decision Transformer, in stochastic games. It introduces Conservative Adversarially Robust Transformer (CART), which relabels offline trajectories with in-sample expected minimax returns-to-go and models the interaction between protagonist and adversary as stage-games to compute NashQ values via expectile regression. By conditioning the transformer on $Q_{\text{CART}}$, CART yields more conservative and robust value estimates in the presence of transition stochasticity, improving worst-case performance compared to DT and ARDT. The results demonstrate improved minimax value estimation and higher worst-case returns in synthetic stochastic games, highlighting CART’s potential for robust offline RL under adversarial and stochastic dynamics.

Abstract

The Transformer, a highly expressive architecture for sequence modeling, has recently been adapted to solve sequential decision-making, most notably through the Decision Transformer (DT), which learns policies by conditioning on desired returns. Yet, the adversarial robustness of reinforcement learning methods based on sequence modeling remains largely unexplored. Here we introduce the Conservative Adversarially Robust Decision Transformer (CART), to our knowledge the first framework designed to enhance the robustness of DT in adversarial stochastic games. We formulate the interaction between the protagonist and the adversary at each stage as a stage game, where the payoff is defined as the expected maximum value over subsequent states, thereby explicitly incorporating stochastic state transitions. By conditioning Transformer policies on the NashQ value derived from these stage games, CART generates policy that are simultaneously less exploitable (adversarially robust) and conservative to transition uncertainty. Empirically, CART achieves more accurate minimax value estimation and consistently attains superior worst-case returns across a range of adversarial stochastic games.

Figures (8)

• Figure 1: An illustrative example of stochastic game (detailed game setup in Figure \ref{['fig:tree']}). ARDT demonstrates overly optimism in estimating values of state and actions, regardless the small probability to reach the desired state $s'_2$. In contrast, our method CART addresses the stochasticity by assigning expected maximum values. Since the value estimation is more conservative and accurate, we call the proposed method Conservative ARDT (CART). We evaluate the returns against the optimal adversary across different methods in the right-hand-side table.
• Figure 2: Return condition $z=Q(s,a)$ in training different DTs. In CART, additional function $V$ is added to address the stochasticity in state transition from $s$ to $s'$.
• Figure 3: LHS demonstrates the worst-case return versus target return plot comparing the proposed CART algorithm against vanilla DT and ARDT, on our two-stage Stochastic Game. RHS represents the average performance Comparison among CART, ARDT, and DT across 5 synthetic adversarial stochastic games where we set high target return during decoding.
• Figure 4: LHS presents the game with a target return of 8 where decision-maker P is confronted by Adversary A. In the worst-case scenario, if P chooses action $a_0$, A will respond with $\bar{a}_0$ to minimize P's expected payoff at the next state, and if P chooses $a_1$ or $a_2$, payoffs are independent of A's strategic behaviors. Consequently, the worst-case expected returns for actions $a_0$, $a_1$, and $a_2$ are 6, 8, and 8, respectively. Therefore, the robust choice of action for the decision-maker is $a_1$ or $a_2$. For training, the data are collected by running uniformly random behavior policy for long enough such that all the trajectories are covered. RHS represents the implementation of ARDT in this scenario. Due to the stochastic nature of the state transition, ARDT can be misled by rare, high-payoff events.
• Figure 5: LHS demonstrates a variant of the two-stage Stochastic Game with a target return of 8 where the rare payoff is adjusted to be 100. RHS demonstrates a variant of the two-stage Stochastic Game with a target return of 8 when the Protagonist chooses action 0 at state 0 and the Adversary chooses action 1, the probability of transition to state 1 and 2 would be 20% and 80%, respectively.