Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NLICE: Synthetic Medical Record Generation for Effective Primary Healthcare Differential Diagnosis

Zaid Al-Ars, Obinna Agba, Zhuoran Guo, Christiaan Boerkamp, Ziyaad Jaber, Tareq Jaber

TL;DR

This work tackles the scarcity of medically accurate public data for primary-care differential diagnosis by generating two synthetic datasets: a SymCat-based dataset and an NLICE-augmented dataset created with Synthea. It systematically evaluates Naive Bayes and Random Forest classifiers on both datasets, showing that NLICE augmentation yields substantially higher diagnostic performance (Top-1 around 82% and Top-5 above 90%) than SymCat alone. The study also demonstrates robustness to realistic perturbations, varying symptom counts, and injected symptoms, with NLICE models showing greater stability and higher confidence in correct predictions. By integrating expert-informed, multi-attribute symptom descriptions with synthetic patient records, the authors present a practical pathway to operationalize AI assistance for differential diagnosis in healthcare, and they release NLICE as open-source software for broader use.

Abstract

This paper offers a systematic method for creating medical knowledge-grounded patient records for use in activities involving differential diagnosis. Additionally, an assessment of machine learning models that can differentiate between various conditions based on given symptoms is also provided. We use a public disease-symptom data source called SymCat in combination with Synthea to construct the patients records. In order to increase the expressive nature of the synthetic data, we use a medically-standardized symptom modeling method called NLICE to augment the synthetic data with additional contextual information for each condition. In addition, Naive Bayes and Random Forest models are evaluated and compared on the synthetic data. The paper shows how to successfully construct SymCat-based and NLICE-based datasets. We also show results for the effectiveness of using the datasets to train predictive disease models. The SymCat-based dataset is able to train a Naive Bayes and Random Forest model yielding a 58.8% and 57.1% Top-1 accuracy score, respectively. In contrast, the NLICE-based dataset improves the results, with a Top-1 accuracy of 82.0% and Top-5 accuracy values of more than 90% for both models. Our proposed data generation approach solves a major barrier to the application of artificial intelligence methods in the healthcare domain. Our novel NLICE symptom modeling approach addresses the incomplete and insufficient information problem in the current binary symptom representation approach. The NLICE code is open sourced at https://github.com/guozhuoran918/NLICE.

NLICE: Synthetic Medical Record Generation for Effective Primary Healthcare Differential Diagnosis

TL;DR

This work tackles the scarcity of medically accurate public data for primary-care differential diagnosis by generating two synthetic datasets: a SymCat-based dataset and an NLICE-augmented dataset created with Synthea. It systematically evaluates Naive Bayes and Random Forest classifiers on both datasets, showing that NLICE augmentation yields substantially higher diagnostic performance (Top-1 around 82% and Top-5 above 90%) than SymCat alone. The study also demonstrates robustness to realistic perturbations, varying symptom counts, and injected symptoms, with NLICE models showing greater stability and higher confidence in correct predictions. By integrating expert-informed, multi-attribute symptom descriptions with synthetic patient records, the authors present a practical pathway to operationalize AI assistance for differential diagnosis in healthcare, and they release NLICE as open-source software for broader use.

Abstract

This paper offers a systematic method for creating medical knowledge-grounded patient records for use in activities involving differential diagnosis. Additionally, an assessment of machine learning models that can differentiate between various conditions based on given symptoms is also provided. We use a public disease-symptom data source called SymCat in combination with Synthea to construct the patients records. In order to increase the expressive nature of the synthetic data, we use a medically-standardized symptom modeling method called NLICE to augment the synthetic data with additional contextual information for each condition. In addition, Naive Bayes and Random Forest models are evaluated and compared on the synthetic data. The paper shows how to successfully construct SymCat-based and NLICE-based datasets. We also show results for the effectiveness of using the datasets to train predictive disease models. The SymCat-based dataset is able to train a Naive Bayes and Random Forest model yielding a 58.8% and 57.1% Top-1 accuracy score, respectively. In contrast, the NLICE-based dataset improves the results, with a Top-1 accuracy of 82.0% and Top-5 accuracy values of more than 90% for both models. Our proposed data generation approach solves a major barrier to the application of artificial intelligence methods in the healthcare domain. Our novel NLICE symptom modeling approach addresses the incomplete and insufficient information problem in the current binary symptom representation approach. The NLICE code is open sourced at https://github.com/guozhuoran918/NLICE.
Paper Structure (25 sections, 8 equations, 6 figures, 4 tables)

This paper contains 25 sections, 8 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Modeling of differential diagnosis
  3. Methods
  4. Synthea
  5. SymCat
  6. NLICE database
  7. NLICE symptom modeling
  8. NLICE data collection
  9. Combining NLICE and Synthea
  10. Selected ML models
  11. Evaluation metrics
  12. Mimicking real-world scenarios
  13. Varying minimum number of symptoms per condition
  14. Perturbing the condition-symptom probabilities
  15. Injecting additional symptoms
  16. ...and 10 more sections

Figures (6)

  • Figure 1: Posterior probability estimate comparisons using Random Forest and Naive Bayes for the SymCat dataset
  • Figure 2: Posterior probability estimate comparisons using Random Forest and Naive Bayes for the NLICE dataset
  • Figure 3: Prediction confidence threshold using Random Forest for the SymCat dataset
  • Figure 4: Prediction confidence threshold using Random Forest for the NLICE dataset
  • Figure 5: Effect of increased minimum symptom expressed per condition on model performance for SymCat data
  • ...and 1 more figures