Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

C-kNN-LSH: A Nearest-Neighbor Algorithm for Sequential Counterfactual Inference

Jing Wang, Jie Shen, Qiaomin Xie, Jeremy C Weiss

TL;DR

This work tackles the challenge of estimating individualized counterfactual trajectories in high-dimensional longitudinal data, exemplified by Long COVID patients. It introduces C-kNN-LSH, a framework that combines latent history compression via a variational encoder guided by an LLM, Locality-Sensitive Hashing-based nearest-neighbor matching in latent space, and a doubly robust correction to estimate $theta_{i,t}(a) = E[Y_{i,t}(a) | H_{i,t}]$. The approach provides consistency guarantees under approximate latent sufficiency and demonstrates superior recovery-heterogeneity capture and policy-value estimation on a real cohort of 13,511 participants, outperforming traditional baselines. The results suggest that probabilistic representation learning can support valid causal reasoning in richly observed sequential biomedical data without explicit causal graph recovery, offering a scalable path toward personalized, data-driven clinical decision-making.

Abstract

Estimating causal effects from longitudinal trajectories is central to understanding the progression of complex conditions and optimizing clinical decision-making, such as comorbidities and long COVID recovery. We introduce \emph{C-kNN--LSH}, a nearest-neighbor framework for sequential causal inference designed to handle such high-dimensional, confounded situations. By utilizing locality-sensitive hashing, we efficiently identify ``clinical twins'' with similar covariate histories, enabling local estimation of conditional treatment effects across evolving disease states. To mitigate bias from irregular sampling and shifting patient recovery profiles, we integrate neighborhood estimator with a doubly-robust correction. Theoretical analysis guarantees our estimator is consistent and second-order robust to nuisance error. Evaluated on a real-world Long COVID cohort with 13,511 participants, \emph{C-kNN-LSH} demonstrates superior performance in capturing recovery heterogeneity and estimating policy values compared to existing baselines.

C-kNN-LSH: A Nearest-Neighbor Algorithm for Sequential Counterfactual Inference

TL;DR

This work tackles the challenge of estimating individualized counterfactual trajectories in high-dimensional longitudinal data, exemplified by Long COVID patients. It introduces C-kNN-LSH, a framework that combines latent history compression via a variational encoder guided by an LLM, Locality-Sensitive Hashing-based nearest-neighbor matching in latent space, and a doubly robust correction to estimate . The approach provides consistency guarantees under approximate latent sufficiency and demonstrates superior recovery-heterogeneity capture and policy-value estimation on a real cohort of 13,511 participants, outperforming traditional baselines. The results suggest that probabilistic representation learning can support valid causal reasoning in richly observed sequential biomedical data without explicit causal graph recovery, offering a scalable path toward personalized, data-driven clinical decision-making.

Abstract

Estimating causal effects from longitudinal trajectories is central to understanding the progression of complex conditions and optimizing clinical decision-making, such as comorbidities and long COVID recovery. We introduce \emph{C-kNN--LSH}, a nearest-neighbor framework for sequential causal inference designed to handle such high-dimensional, confounded situations. By utilizing locality-sensitive hashing, we efficiently identify ``clinical twins'' with similar covariate histories, enabling local estimation of conditional treatment effects across evolving disease states. To mitigate bias from irregular sampling and shifting patient recovery profiles, we integrate neighborhood estimator with a doubly-robust correction. Theoretical analysis guarantees our estimator is consistent and second-order robust to nuisance error. Evaluated on a real-world Long COVID cohort with 13,511 participants, \emph{C-kNN-LSH} demonstrates superior performance in capturing recovery heterogeneity and estimating policy values compared to existing baselines.
Paper Structure (24 sections, 20 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 24 sections, 20 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Challenges
  3. Problem Setup
  4. LLM-Augmented History Compression
  5. Local Counterfactual Estimation
  6. Doubly Robust Correction
  7. Neural Network Architecture
  8. Theoretical Discussion
  9. Latent sufficiency and approximation error.
  10. Consistency
  11. Discussion.
  12. Related Works
  13. Experiments
  14. Dataset
  15. History Construction
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Estimated causal effects $\hat{\theta}(a)$ across treatment actions for three phenotypes under a 30-day history lookback. Each curve corresponds to a different concept set, including the baseline ALL representation. Shaded regions indicate variability across individuals within each phenotype.
  • Figure 2: Estimated causal effects $\hat{\theta}(a)$ across treatment actions for three phenotypes under a 180-day history lookback. Each curve corresponds to a different concept set, including the baseline ALL representation. Shaded regions indicate variability across individuals within each phenotype.
  • Figure 3: Differences in estimated causal effects relative to the baseline ALL representation ($\hat{\theta}_c-\hat{\theta}_{\text{ALL}}$) for phenotype 0 under a 30-day lookback. Positive values indicate higher estimated effects compared to the baseline.
  • Figure 4: Differences in estimated causal effects relative to the baseline ALL representation ($\hat{\theta}_c-\hat{\theta}_{\text{ALL}}$) for phenotype 0 under a 180-day lookback. Positive values indicate higher estimated effects compared to the baseline.