From algorithms to action: improving patient care requires causality

TL;DR

The paper argues that predictive accuracy alone is insufficient for guiding cancer treatment decisions because it ignores causal effects of treatment policies. It advocates a causal framework and prediction-under-intervention approaches to evaluate the decision value of models. It discusses validation methods including cluster randomized trials and analysis of RCT data to estimate outcomes under proposed policies, and highlights DAG-based unconfoundedness and sensitivity analyses. The work emphasizes aligning predictive tools with patient-centered decision-making to truly improve outcomes.

Abstract

In cancer research there is much interest in building and validating outcome predicting outcomes to support treatment decisions. However, because most outcome prediction models are developed and validated without regard to the causal aspects of treatment decision making, many published outcome prediction models may cause harm when used for decision making, despite being found accurate in validation studies. Guidelines on prediction model validation and the checklist for risk model endorsement by the American Joint Committee on Cancer do not protect against prediction models that are accurate during development and validation but harmful when used for decision making. We explain why this is the case and how to build and validate models that are useful for decision making.

Figures (4)

• Figure 1: Illustration of the use of outcome prediction models that ignore treatment allocations in the historical data (i.e. are treatment naive) for treatment decision making. These models change the treatment decisions and thus patient outcomes but whether this change improves patient outcomes is not determined by the prediction accuracy of the outcome prediction model.
• Figure 2: Illustration of the difference between outcome prediction model accuracy and its value for treatment decision making. Validation of an outcome prediction model following the checklist leads to a reliable estimate of the outcome prediction model's accuracy. However, because the outcome prediction model relies on a fixed historic treatment policy, prediction accuracy does not imply value for decision making, as visualized with the gap. This gap can only be bridged with causality.
• Figure 3: Simplified Directed Acyclic Graph for the decision between surgery and radiotherapy for overall survival in lung cancer patients. As an example, consider a hypothetical study in early-stage lung cancer where researchers investigate whether the relative effectiveness of surgery versus radiotherapy for overall survival depends on a certain single-nucleotide polymorphism (SNP). The SNP assay was performed for the study only so this information did not affect the treatment decision. A with four variables for this study is presented in this Figure. In this , the variables age and SNP both have arrows to overall survival, but only age influences the treatment decision as older patients are less likely to get surgery. The from indicates that unconfoundedness holds when age is conditioned on in the analysis, as age is the only confounder between the treatment and the outcome pearl_causality_2009.
• Figure 4: Flowchart of what to do depending on the costliness of cluster randomized controlled trials. Costliness of cluster should be taken broadly, including time, money and ethical considerations.