Benchmarking covariate-adjustment strategies for randomized clinical trials

TL;DR

A large-scale empirical benchmarking using individual-level data from 50 publicly accessible RCTs provides the first large-scale empirical evidence that transparent and parsimonious covariate adjustment is sufficient and often preferable for routine RCT analysis.

Abstract

Covariate adjustment is widely recommended to improve statistical efficiency in randomized clinical trials (RCTs), yet empirical evidence comparing available strategies remains limited. This lack of real-world evaluation leaves unresolved practical questions about which adjustment methods to use and which covariates to include. To address this gap, we conduct a large-scale empirical benchmarking using individual-level data from 50 publicly accessible RCTs comprising 29,094 participants and 574 treatment-outcome pairs. We evaluate 18 analytical strategies formed by combining six estimators-including classical regression, inverse probability weighting, and machine-learning methods-with three covariate-selection rules. Across diverse therapeutic areas, covariate adjustment consistently improves precision, yielding median variance reductions of 13.3% relative to unadjusted analyses for continuous outcomes and 4.6% for binary outcomes. However, machine-learning algorithms implemented with default hyperparameter settings do not yield efficiency gains beyond simple linear models. Parsimonious regression approaches, such as analysis of covariance, deliver stable, reproducible performance even in moderate sample sizes. Together, these findings provide the first large-scale empirical evidence that transparent and parsimonious covariate adjustment is sufficient and often preferable for routine RCT analysis. All curated datasets and analysis code are openly released as a reproducible benchmark resource to support future clinical research and methodological development.

Figures (5)

• Figure 1: Workflow of the benchmarking study. Panel A displays the data characteristics. Panel B presents the covariate adjustment methods for benchmarking. Panel C demonstrates the performance metrics.
• Figure 2: The summary results of the method comparison for continuous outcomes (blue) and binary outcomes (red). For comparing precision, the box plot of PVR is given, with larger values representing better performance. The box plot for estimate shift measures the scaled difference in point estimates between adjusted and unadjusted analyses. CAG and CAL measure the real-world gain and loss from covariate adjustment, respectively. Error rate is the proportion of analyses for which R reports errors.
• Figure 3: Proportional variance reduction varied by sample sizes on the log10 scale. Curves are generated by locally estimated scatterplot smoothing. Solid and dashed curves represent regression adn machine-learning-based estimators, respectively.
• Figure 4: Performance of machine-learning algorithms with TMLE adjusting for all covariates. For comparing precision, the box plot of PVR is given, with larger values representing better performance.
• Figure 5: Practical recommendations for covariate adjustment.