Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-To-End Causal Effect Estimation from Unstructured Natural Language Data

Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul G. Krishnan, Chris J. Maddison

TL;DR

It is shown how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions, and NATURAL is a first step towards an automated pipeline to tap this resource.

Abstract

Knowing the effect of an intervention is critical for human decision-making, but current approaches for causal effect estimation rely on manual data collection and structuring, regardless of the causal assumptions. This increases both the cost and time-to-completion for studies. We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions. We introduce NATURAL, a novel family of causal effect estimators built with LLMs that operate over datasets of unstructured text. Our estimators use LLM conditional distributions (over variables of interest, given the text data) to assist in the computation of classical estimators of causal effect. We overcome a number of technical challenges to realize this idea, such as automating data curation and using LLMs to impute missing information. We prepare six (two synthetic and four real) observational datasets, paired with corresponding ground truth in the form of randomized trials, which we used to systematically evaluate each step of our pipeline. NATURAL estimators demonstrate remarkable performance, yielding causal effect estimates that fall within 3 percentage points of their ground truth counterparts, including on real-world Phase 3/4 clinical trials. Our results suggest that unstructured text data is a rich source of causal effect information, and NATURAL is a first step towards an automated pipeline to tap this resource.

End-To-End Causal Effect Estimation from Unstructured Natural Language Data

TL;DR

It is shown how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions, and NATURAL is a first step towards an automated pipeline to tap this resource.

Abstract

Knowing the effect of an intervention is critical for human decision-making, but current approaches for causal effect estimation rely on manual data collection and structuring, regardless of the causal assumptions. This increases both the cost and time-to-completion for studies. We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions. We introduce NATURAL, a novel family of causal effect estimators built with LLMs that operate over datasets of unstructured text. Our estimators use LLM conditional distributions (over variables of interest, given the text data) to assist in the computation of classical estimators of causal effect. We overcome a number of technical challenges to realize this idea, such as automating data curation and using LLMs to impute missing information. We prepare six (two synthetic and four real) observational datasets, paired with corresponding ground truth in the form of randomized trials, which we used to systematically evaluate each step of our pipeline. NATURAL estimators demonstrate remarkable performance, yielding causal effect estimates that fall within 3 percentage points of their ground truth counterparts, including on real-world Phase 3/4 clinical trials. Our results suggest that unstructured text data is a rich source of causal effect information, and NATURAL is a first step towards an automated pipeline to tap this resource.
Paper Structure (28 sections, 24 equations, 11 figures, 6 tables)

This paper contains 28 sections, 24 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Inverse Propensity Score Weighting (IPW).
  5. Outcome Imputation (OI).
  6. NATURAL estimators of the ATE
  7. Implementing NATURAL estimators with Large Language Models
  8. Limitations and Broader Impact
  9. Empirical Evaluation
  10. Synthetic Datasets.
  11. Real-world Datasets.
  12. How well does NATURAL estimate observational distributions from self-reported data?
  13. How do NATURAL methods compare to one another and to trained baselines?
  14. How do different choices in the NATURAL pipeline effect ATE prediction?
  15. How well do different estimates of propensity score balance covariates?
  16. ...and 13 more sections

Figures (11)

  • Figure 1: When compared to experimental and other observational studies, NATURAL has lower costs and provides greater diversity in cohort selection, for causal effect estimation.
  • Figure 2: Our pipeline leverages LLMs to curate data that can be plugged into natural language conditioned estimators for average treatment effects.
  • Figure 3: For Hillstrom (left) and Retail Hero (right), the KL divergence between estimated joint and propensity distributions and their true counterparts reduces with increasing number of posts (top), as does the RMSE between the NATURAL Full estimate and true ATE (bottom).
  • Figure 4: Ablation study on Semaglutide vs. Tirzepatide, to tease apart the effect of data filtering and imputation (left) as well as LLM scale for conditionals (right) on NATURAL performance.
  • Figure 5: NATURAL propensity scores balance the Semaglutide vs. Tirzepatide covariates better than uniform scores.
  • ...and 6 more figures