Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can LLMs Assist Expert Elicitation for Probabilistic Causal Modeling?

Olha Shaposhnyk, Daria Zahorska, Svetlana Yanushkevich

TL;DR

The paper investigates using Large Language Models (LLMs) as a tool for expert elicitation in constructing probabilistic causal models, specifically Bayesian networks, within smart-health applications. By applying LLM-generated BN structures to a Sleep Health and Lifestyle dataset and comparing against BIC-driven and human-expert graphs, the study uses SEM validation and entropy measures to assess structure quality. Results indicate that LLM-based BNs tend to exhibit lower entropy (i.e., higher confidence) and fewer logical inconsistencies than BIC-driven graphs, though challenges such as hallucinations and small sample size remain. Overall, the work demonstrates a promising, transparent, and scalable approach to eliciting causal knowledge with LLMs, while calling for further validation and bias mitigation in real-world deployments.

Abstract

Objective: This study investigates the potential of Large Language Models (LLMs) as an alternative to human expert elicitation for extracting structured causal knowledge and facilitating causal modeling in biometric and healthcare applications. Material and Methods: LLM-generated causal structures, specifically Bayesian networks (BNs), were benchmarked against traditional statistical methods (e.g., Bayesian Information Criterion) using healthcare datasets. Validation techniques included structural equation modeling (SEM) to verifying relationships, and measures such as entropy, predictive accuracy, and robustness to compare network structures. Results and Discussion: LLM-generated BNs demonstrated lower entropy than expert-elicited and statistically generated BNs, suggesting higher confidence and precision in predictions. However, limitations such as contextual constraints, hallucinated dependencies, and potential biases inherited from training data require further investigation. Conclusion: LLMs represent a novel frontier in expert elicitation for probabilistic causal modeling, promising to improve transparency and reduce uncertainty in the decision-making using such models.

Can LLMs Assist Expert Elicitation for Probabilistic Causal Modeling?

TL;DR

The paper investigates using Large Language Models (LLMs) as a tool for expert elicitation in constructing probabilistic causal models, specifically Bayesian networks, within smart-health applications. By applying LLM-generated BN structures to a Sleep Health and Lifestyle dataset and comparing against BIC-driven and human-expert graphs, the study uses SEM validation and entropy measures to assess structure quality. Results indicate that LLM-based BNs tend to exhibit lower entropy (i.e., higher confidence) and fewer logical inconsistencies than BIC-driven graphs, though challenges such as hallucinations and small sample size remain. Overall, the work demonstrates a promising, transparent, and scalable approach to eliciting causal knowledge with LLMs, while calling for further validation and bias mitigation in real-world deployments.

Abstract

Objective: This study investigates the potential of Large Language Models (LLMs) as an alternative to human expert elicitation for extracting structured causal knowledge and facilitating causal modeling in biometric and healthcare applications. Material and Methods: LLM-generated causal structures, specifically Bayesian networks (BNs), were benchmarked against traditional statistical methods (e.g., Bayesian Information Criterion) using healthcare datasets. Validation techniques included structural equation modeling (SEM) to verifying relationships, and measures such as entropy, predictive accuracy, and robustness to compare network structures. Results and Discussion: LLM-generated BNs demonstrated lower entropy than expert-elicited and statistically generated BNs, suggesting higher confidence and precision in predictions. However, limitations such as contextual constraints, hallucinated dependencies, and potential biases inherited from training data require further investigation. Conclusion: LLMs represent a novel frontier in expert elicitation for probabilistic causal modeling, promising to improve transparency and reduce uncertainty in the decision-making using such models.

Paper Structure

This paper contains 17 sections, 4 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Problem Formulation and Contribution
  4. Methodology
  5. Data selection
  6. BN Construction
  7. Expert elicitation for constructing BN
  8. Causal Modeling
  9. Causal graph structure created by human experts
  10. Causal graph structure based on information criteria
  11. Causal graph structure based on LLM
  12. Comparison of entropy across BN generation methods
  13. Example of using BN for Decision Support
  14. Creating BN
  15. Examples of case-study on the BN
  16. ...and 2 more sections

Figures (6)

  • Figure 1: Causal modeling workflow: data preprocessing, building causal graph using LLM and information criteria, and constructing a BN. The novelty of this work such as applying the LLM-based expert elicitation is highlighted in green.
  • Figure 2: Causal graph structure derived from human expertise. Node color indicates entropy values computed per node, while arc thickness represents mutual information calculated for each connection.
  • Figure 3: Causal graph structure derived from BIC. Node color indicates entropy values computed per node, while arc thickness represents mutual information calculated for each connection.
  • Figure 4: Causal graph structure derived from LLM. Node color indicates entropy values computed per node, while arc thickness represents mutual information calculated for each connection.
  • Figure 5: BN generated using LLM expert elicitation and used as a decision support tool, which represents the relationship between nodes
  • ...and 1 more figures