Salvaging Forbidden Treasure in Medical Data: Utilizing Surrogate Outcomes and Single Records for Rare Event Modeling

TL;DR

The paper introduces HiRRR, a Hybrid & Integrative Reduced Rank Regression framework that jointly models a primary rare outcome with surrogate concurrent outcomes while incorporating single-record data through a shared latent representation. By combining supervised learning on multi-record data with unsupervised latent learning from single-record data, HiRRR leverages a low-rank encoder–decoder structure to improve both estimation and, to a lesser extent, prediction for rare health events. The authors provide algorithmic solutions via blockwise coordinate descent, prove non-asymptotic subspace error bounds showing that single-record data effectively augment sample size, and validate the approach with simulations. Applied to Connecticut HIDD data for pediatric-to-young-adult suicide risk, HiRRR improves risk-factor estimation, increases detection of high-risk individuals, and demonstrates the practical value of surrogates and single-encounter information for rare-event modeling in healthcare.

Abstract

The vast repositories of Electronic Health Records (EHR) and medical claims hold untapped potential for studying rare but critical events, such as suicide attempt. Conventional setups often model suicide attempt as a univariate outcome and also exclude any ``single-record'' patients with a single documented encounter due to a lack of historical information. However, patients who were diagnosed with suicide attempts at the only encounter could, to some surprise, represent a substantial proportion of all attempt cases in the data, as high as 70--80%. We innovate a hybrid and integrative learning framework to leverage concurrent outcomes as surrogates and harness the forbidden yet precious information from single-record data. Our approach employs a supervised learning component to learn the latent variables that connect primary (e.g., suicide) and surrogate outcomes (e.g., mental disorders) to historical information. It simultaneously employs an unsupervised learning component to utilize the single-record data, through the shared latent variables. As such, our approach offers a general strategy for information integration that is crucial to modeling rare conditions and events. With hospital inpatient data from Connecticut, we demonstrate that single-record data and concurrent diagnoses indeed carry valuable information, and utilizing them can substantially improve suicide risk modeling.

Figures (6)

• Figure 1: Schematic representation of the hybrid & integrative learning framework.
• Figure 2: Cohort selection and case-control setup with HIDD data.
• Figure 3: Application: Boxplots of the performance metrics from the random splitting procedure.
• Figure 4: Simulation: Estimation performance of $\mathrm{Er}(\hbox{\bf V}_C)$ and $\mathrm{Er}(\hbox{\bf C})$ at $q=30, r=10$, and $b=0.01$ for continuous-outcome scenarios.
• Figure 5: Simulation: Estimation performance of $\mathrm{Er}(\hbox{\bf V}_C)$ and $\mathrm{Er}(\hbox{\bf C})$ at $q=100, r=10$, and $b=0.05$ for binary-outcome scenarios (GLM results were not presented as they fell outside the range).

Theorems & Definitions (4)

• Proposition 4.1
• Theorem 4.2: Non-Asymptotic Subspace Error
• Lemma D.1: Matrix Bernstein Inequality
• Lemma D.2: Operator Norm of Sub-Gaussian Vector