Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learning

TL;DR

The paper addresses sequential policy evaluation in time-series A/B testing, where carryover and long-range dependencies hinder traditional designs. It introduces Transformer RL (TRL), which encodes the entire experimental history with a transformer to form the state $S_t$ and uses a double deep Q-network to choose actions with the objective of minimizing $\mathrm{MSE}(\pi)$ of the ATE estimator. A central impossibility theorem shows that the optimal allocation generally depends on the full history, establishing a fundamental limitation of history-free designs under doubly robust estimation. Empirical results across synthetic data, a publicly available dispatch simulator, and a real ridesharing dataset demonstrate that TRL reduces the ATE estimator MSE relative to existing designs. The framework offers a model-free, history-aware approach to efficient online experimentation in dynamic, carryover-prone environments.

Abstract

A/B testing has become a gold standard for modern technological companies to conduct policy evaluation. Yet, its application to time series experiments, where policies are sequentially assigned over time, remains challenging. Existing designs suffer from two limitations: (i) they do not fully leverage the entire history for treatment allocation; (ii) they rely on strong assumptions to approximate the objective function (e.g., the mean squared error of the estimated treatment effect) for optimizing the design. We first establish an impossibility theorem showing that failure to condition on the full history leads to suboptimal designs, due to the dynamic dependencies in time series experiments. To address both limitations simultaneously, we next propose a transformer reinforcement learning (RL) approach which leverages transformers to condition allocation on the entire history and employs RL to directly optimize the MSE without relying on restrictive assumptions. Empirical evaluations on synthetic data, a publicly available dispatch simulator, and a real-world ridesharing dataset demonstrate that our proposal consistently outperforms existing designs.

Figures (11)

• Figure 1: Illustration of the proposed transformer reinforcement learning algorithm. Our algorithm employs a transformer encoder to summarize the full historical context and to produce the state $\{S_t\}_t$. These states are then fed into a double deep Q-network agent, which outputs an optimal policy that minimizes the mean squared error of the ATE estimator (encoded as the (negative) reward $\{R_t\}_t$).
• Figure 2: Graphical illustrations of treatment allocations for existing designs (a)–(d) and our proposed design (e). Specifically, existing designs condition treatment allocation on only the current observation (a), the initial action (b), a limited history (c), or the time index (d). In contrast, the proposed design conditions on the entire historical information (e).
• Figure 3: Barplots of empirical MSEs under different designs with their confidence intervals in the synthetic environment, across varying variances (Setting (i)) and transition structures (Setting (ii)).
• Figure 4: Barplots of the empirical MSEs under different designs in the real-data-based simulation, with a $5\%$ performance improvement from the new policy.
• Figure 5: Barplots of empirical MSEs under different designs with their confidence intervals in the dispatch environment, across different days.

Theorems & Definitions (2)

• Theorem 1: Impossibility theorem
• proof