General Demographic Foundation Models for Enhancing Predictive Performance Across Diseases and Populations

TL;DR

This work introduces the General Demographic Pre-trained (GDP) model, a foundation model dedicated to learning representations of demographic attributes—specifically age and gender—from tabular EHR data. Trained in a self-supervised fashion to predict Charlson Comorbidity Index using about two million records, GDP yields a 32-dimensional embedding $32$ that is transferred to downstream tasks (pneumonia, osteoporosis, thyroid disease) via LightGBM to assess generalization across diseases and populations. Across datasets, sequential input ordering (Seq) consistently improves discrimination ($AUROC$) and calibration ($ECE$) when age/gender signals are predictive, and it increases the relative information gain contributed by demographic features, even in less favorable settings. The results suggest that demographic-focused foundation models can augment predictive performance and should be integrated with other modalities to maximize clinical relevance and transferability across diverse populations and diseases.

Abstract

Demographic attributes are universally present in electronic health records. They are the most widespread information across populations and diseases, and serve as vital predictors in clinical risk stratification and treatment decisions. Despite their significance, these attributes are often treated as auxiliaries in model design, with limited attention being paid to learning their representations. This study explored the development of a General Demographic Pre-trained (GDP) model as a foundational model tailored to demographic attributes, focusing on age and gender. The model is pre-trained and evaluated using datasets with diverse diseases and populations compositions from different geographic regions. The composition of GDP architecture was explored through examining combinations of ordering approaches and encoding methods to transform tabular demographic inputs into effective latent embeddings. Results demonstrate the feasibility of GDP to generalize across task, diseases, and populations. In detailed composition, the sequential ordering substantially improves model performance in discrimination, calibration, and the corresponding information gain at each decision tree split, particularly in diseases where age and gender contribute significantly to risk stratification. Even in datasets where demographic attributes hold relatively low predictive value, GDP enhances the representational importance, increasing their influence in downstream gradient boosting models. The findings suggest that foundation models for tabular demographic attributes offer a promising direction for improving predictive performance in healthcare applications.

Figures (5)

• Figure 1: General demographical pre-trained model training and validation flow. GDP: General Demographic Pre-trained; NHIRD: National Health Insurance Research Database of Taiwan; $NS$: non-sequential ordering; $Seq$: Sequential ordering; $\emph{trad}$: traditional encoding; $\emph{PE}$: positional encoding; $\emph{txt}$: text-based semantic encoding.
• Figure 2: Representation distribution of pneumonia dataset.
• Figure 3: Representation distribution of osteoporosis dataset.
• Figure 4: Representation distribution of thyroid dataset.
• Figure 5: Comparison of age and gender information gain across approaches. Scores reflect the aggregate contribution of age and gender, divided by the overall information gain of all features.