Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mind the data gap: Missingness Still Shapes Large Language Model Prognoses

Yuta Kobayashi, Vincent Jeanselme, Shalmali Joshi

TL;DR

The paper investigates how informative missingness in healthcare data shapes the zero-shot prognostic performance of large language models. By analyzing 10 LLMs across three clinical tasks on MIMIC-IV and an external CUMC dataset, it shows that missingness representations and prompting can strongly affect predictions and calibration in a model- and task-dependent manner, with larger models generally benefiting from explicit missingness cues. A theoretical decomposition clarifies why these effects arise, linking decoding prompts, prior knowledge, and Bayes-optimal reasoning. The findings call for systematic evaluation and transparent reporting of missing-data representations in LLM deployments, highlighting practical implications for reliable, healthcare-facing AI systems and prompting further work in prompt design and model finetuning.

Abstract

Data collection often reflects human decisions. In healthcare, for instance, a referral for a diagnostic test is influenced by the patient's health, their preferences, available resources, and the practitioner's recommendations. Despite the extensive literature on the informativeness of missingness, its implications on the performance of Large Language Models (LLMs) have not been studied. Through a series of experiments on data from Columbia University Medical Center, a large urban academic medical center, and MIMIC-IV, we demonstrate that patterns of missingness significantly impact zero-shot predictive performance. Notably, the explicit inclusion of missingness indicators at prompting benefits some while hurting other LLMs' zero-shot predictive performance and calibration, suggesting an inconsistent impact. The proposed aggregated analysis and theoretical insights suggest that larger models benefit from these interventions, while smaller models can be negatively impacted. The LLM paradigm risks obscuring the impact of missingness, often neglected even in conventional ML, even further. We conclude that there is a need for more transparent accounting and systematic evaluation of the impact of representing (informative) missingness on downstream performance.

Mind the data gap: Missingness Still Shapes Large Language Model Prognoses

TL;DR

The paper investigates how informative missingness in healthcare data shapes the zero-shot prognostic performance of large language models. By analyzing 10 LLMs across three clinical tasks on MIMIC-IV and an external CUMC dataset, it shows that missingness representations and prompting can strongly affect predictions and calibration in a model- and task-dependent manner, with larger models generally benefiting from explicit missingness cues. A theoretical decomposition clarifies why these effects arise, linking decoding prompts, prior knowledge, and Bayes-optimal reasoning. The findings call for systematic evaluation and transparent reporting of missing-data representations in LLM deployments, highlighting practical implications for reliable, healthcare-facing AI systems and prompting further work in prompt design and model finetuning.

Abstract

Data collection often reflects human decisions. In healthcare, for instance, a referral for a diagnostic test is influenced by the patient's health, their preferences, available resources, and the practitioner's recommendations. Despite the extensive literature on the informativeness of missingness, its implications on the performance of Large Language Models (LLMs) have not been studied. Through a series of experiments on data from Columbia University Medical Center, a large urban academic medical center, and MIMIC-IV, we demonstrate that patterns of missingness significantly impact zero-shot predictive performance. Notably, the explicit inclusion of missingness indicators at prompting benefits some while hurting other LLMs' zero-shot predictive performance and calibration, suggesting an inconsistent impact. The proposed aggregated analysis and theoretical insights suggest that larger models benefit from these interventions, while smaller models can be negatively impacted. The LLM paradigm risks obscuring the impact of missingness, often neglected even in conventional ML, even further. We conclude that there is a need for more transparent accounting and systematic evaluation of the impact of representing (informative) missingness on downstream performance.

Paper Structure

This paper contains 36 sections, 2 theorems, 10 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related work: Informative missingness and LLM predictions
  3. Experiment Design: Evaluating impact of missingness encoding on LLM predictions
  4. Experiment design
  5. Models considered.
  6. Tasks.
  7. Prompts.
  8. Evaluation.
  9. Results
  10. Logistic regression shows the informativeness of missingness.
  11. Impact of representing informative missingness is inconsistent across LLMs.
  12. Empirical and theoretical insights
  13. Empirical regression
  14. Missingness scaling law.
  15. Theoretical insights
  16. ...and 21 more sections

Key Result

Theorem 1

For any predictor $f^\pi$ induced by a decoding policy $\pi$, Here $\mathcal{L}_{X,M}(f^\pi)$ and $\mathcal{L}_X(f^\pi)$ denote the expected loss under $P_{X,M}$ and $P_X$, respectively, and $\epsilon^*$ is the difference between the Bayes optimal expected loss under $P_{X,M}$ and $P_X$. $q$ denotes the best predictor achievable zero-shot from the LLM's knowle

Figures (7)

  • Figure 1: Our work shows that explicit instructions to consider missing information affect predictive performance and verbalized uncertainty differently across models and clinical prognoses tasks.
  • Figure 2: LLMs calibration performance with Indicator and with Dropped missingness on MIMIC-IV. The dashed lines reflect the baseline performances using a logistic regression model. Points above the diagonal indicate that the intervention increased LLM's uncertainty. The radius of the data point indicates model size. The impact of encoding and prompting to reason about missingness is task- and model-dependent.
  • Figure 3: Serialization example of a patient showing two strategies: (i) serialization with missing features dropped, and (ii) serialization with explicit missingness
  • Figure 4: Example generation for serialization with missingness with the GPT-OSS 20B model.
  • Figure 5: LLMs predictive performance with Indicator and with Dropped missingness on MIMIC-IV. The dashed lines reflect the baseline performances using a logistic regression model.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1: Error decomposition
  • Theorem 1: Error decomposition