Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accounting for Heavy Censoring in Evaluating the Risk Stratification Abilities of Existing Models for Time to Diagnosis of Huntington Disease

Kyle F. Grosser, Abigail G. Foes, Stellen Li, Vraj Parikh, Tanya P. Garcia, Sarah C. Lotspeich

Abstract

Huntington disease (HD) is a neurodegenerative disease with progressively worsening symptoms. Accurately modeling time to HD diagnosis is essential for clinical trial design. Langbehn's model, the CAG-Age Product (CAP) model, the Prognostic Index Normed (PIN) model, and the Multivariate Risk Score (MRS) model have all been proposed for this task. However, these models may yield conflicting predictions and few studies have systematically compared their performance. Further, those that have could be misleading due to testing the models on the same data used to train them and failing to account for high rates of right censoring (80%+) in performance metrics. We discuss the theoretical foundations of these models, offering intuitive comparisons about their practical feasibility. We externally validate their risk stratification abilities using data from the ENROLL-HD study and two censoring-appropriate performance metrics, guiding model selection for HD clinical trial design. As these models were developed in HD studies that ended more than a decade ago, we compared their predictive performance using published parameters versus updated ones (re-estimated using ENROLL-HD). We show how these models can be used to estimate sample sizes for an HD clinical trial. Based on either metric and using published or updated parameters, the MRS model, which incorporates the most covariates, performed best. However, the simpler PIN model offered similarly good performance while requiring fewer variables, many of which would require patients to undergo additional tests. In illustrating an HD clinical trial design, we defined an optimal threshold based on model performance metrics to determine which patients are more likely to be diagnosed. Sample size calculations using an optimal threshold based on metrics that did not account for censoring, as in previous studies, are shown to lead to underpowered trials.

Accounting for Heavy Censoring in Evaluating the Risk Stratification Abilities of Existing Models for Time to Diagnosis of Huntington Disease

Abstract

Huntington disease (HD) is a neurodegenerative disease with progressively worsening symptoms. Accurately modeling time to HD diagnosis is essential for clinical trial design. Langbehn's model, the CAG-Age Product (CAP) model, the Prognostic Index Normed (PIN) model, and the Multivariate Risk Score (MRS) model have all been proposed for this task. However, these models may yield conflicting predictions and few studies have systematically compared their performance. Further, those that have could be misleading due to testing the models on the same data used to train them and failing to account for high rates of right censoring (80%+) in performance metrics. We discuss the theoretical foundations of these models, offering intuitive comparisons about their practical feasibility. We externally validate their risk stratification abilities using data from the ENROLL-HD study and two censoring-appropriate performance metrics, guiding model selection for HD clinical trial design. As these models were developed in HD studies that ended more than a decade ago, we compared their predictive performance using published parameters versus updated ones (re-estimated using ENROLL-HD). We show how these models can be used to estimate sample sizes for an HD clinical trial. Based on either metric and using published or updated parameters, the MRS model, which incorporates the most covariates, performed best. However, the simpler PIN model offered similarly good performance while requiring fewer variables, many of which would require patients to undergo additional tests. In illustrating an HD clinical trial design, we defined an optimal threshold based on model performance metrics to determine which patients are more likely to be diagnosed. Sample size calculations using an optimal threshold based on metrics that did not account for censoring, as in previous studies, are shown to lead to underpowered trials.

Paper Structure

This paper contains 24 sections, 6 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Huntington Disease, Genetic Testing, and Observational Studies
  3. Designing Clinical Trials: Traditional Versus Preventative
  4. Comparing Existing Models for Time to Diagnosis
  5. Overview
  6. Methods
  7. Notation
  8. Time-to-Diagnosis Models
  9. Model 1: Langbehn
  10. Model 2: CAG-Age Product (CAP)
  11. Model 3: Multivariate Risk Score (MRS)
  12. Model 4: Prognostic Index Normed (PIN)
  13. Modeling Trade-Offs
  14. External Validation Versus Model Updating
  15. Metrics for Evaluating Risk Stratification
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Time-specific Uno's C statistics by model and evaluation time for both published parameters (A) and updated parameters (B), with statistic given by mean from $5$-fold cross-validation for updated parameters. The average Uno's C statistic is a weighted average of the time-specific values. We considered four time-to-diagnosis models: CAG-Age Product (CAP), Langbehn, Multivariate Risk Score (MRS), and Prognostic Index Normed (PIN).
  • Figure 2: Time-specific area under the curve (AUC) of the receiver operating characteristic curve by model and evaluation time for both published parameters (A) and updated parameters (B). We considered four time-to-diagnosis models: Langbehn, CAG-Age Product (CAP), Multivariate Risk Score (MRS), and Prognostic Index Normed (PIN).