Time After Time: Deep-Q Effect Estimation for Interventions on When and What to do

TL;DR

The paper tackles estimating the causal impact of both when to intervene and what intervention to apply under irregular timing. It introduces EDQ, a model-free deep-Q method for decision point processes that dynamically reasons about earliest disagreement times, enabling off-policy evaluation with transformers in continuous time. The authors provide identifiability conditions via local independences and prove an empirical consistency result for EDQ, validating the approach on time-to-failure and tumor-growth simulations. This work advances scalable, high-dimensional causal inference for sequential treatments with irregular observation intervals and offers a practical framework for healthcare, robotics, and finance applications.

Abstract

Problems in fields such as healthcare, robotics, and finance requires reasoning about the value both of what decision or action to take and when to take it. The prevailing hope is that artificial intelligence will support such decisions by estimating the causal effect of policies such as how to treat patients or how to allocate resources over time. However, existing methods for estimating the effect of a policy struggle with \emph{irregular time}. They either discretize time, or disregard the effect of timing policies. We present a new deep-Q algorithm that estimates the effect of both when and what to do called Earliest Disagreement Q-Evaluation (EDQ). EDQ makes use of recursion for the Q-function that is compatible with flexible sequence models, such as transformers. EDQ provides accurate estimates under standard assumptions. We validate the approach through experiments on survival time and tumor growth tasks.

Figures (6)

• Figure 1: A summary of EDQ. Conditional expectations of the outcome, $\mathbb{E}_{P}[Y \vert {\mathcal{H}}_t]$, are estimated by $Q_t({\mathcal{H}}_t; \boldsymbol{\theta})$. Given an observed trajectory ${\mathcal{H}}_{i}$ sampled under the training policy $\lambda_{\mathrm{obs}}$ (bottom trajectory), we fit values $Q_t({\mathcal{H}}_{i,t}; \boldsymbol{\theta})$ by regressing them on a label $\hat{y} := Q_{t+\delta}(\widetilde{{\mathcal{H}}}_{i, t+{\delta}})$ determined by a "counterfactual" trajectory $\widetilde{{\mathcal{H}}}$ sampled from the target policy $\lambda$. $\delta$ is the earliest disagreement time between the observed and counterfactual trajectories. It is either the time of the next observed treatment (counterfactual $1$) or that of the treatment under the target policy (counterfactual $2$).
• Figure 2: The assumed local independence graph for a decision point processes, where our estimand is identifiable from observed data $(N^x, N^a, N^y)$.
• Figure 3: Left. An example of a trajectory from our simulation. The blue curve denotes the value of the vital $x_t$ and red dots mark treatment times. Right. Normalized RMSE under the different simulation settings. The mean is taken over all points in the history of patients in the test data. Rows colored blue have $\lambda_{\mathrm{obs}}=\lambda_{\mathrm{int}}$, and we expect all methods to perform well since train and test distributions match. Red rows are those where the effect of an intervention needs to be estimated.
• Figure 4: Left. Normalized RMSE on the tumor-growth simulation. All methods are affected by distribution shift. EDQ is the most robust out of the methods considered. Right. Normalized RMSE for time-to-failure simulation on short trajectories.
• Figure : Fitted Q-Evaluation (discrete time)

Theorems & Definitions (21)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Theorem 1
• Corollary 1
• Lemma 1
• proof
• Lemma 2
• Theorem 1