An AI Implementation Science Study to Improve Trustworthy Data in a Large Healthcare System

TL;DR

Addresses how to improve trustworthiness and implementation of AI in a large multisite pediatric system. Proposes a concrete modernization of an existing RDW to OMOP CDM v5.4 in MS Fabric, plus a Python-based DQD extension that incorporates METRIC/TAI principles. Compares systematic infrastructure-driven approaches with a CFM-specific, FHIR-enabled case study to evaluate data harmonization and AI performance. Demonstrates partial data-quality gains, highlights practical FHIR-implementation challenges, and argues for hybrid, governance-aware AI implementation in real-world healthcare.

Abstract

The rapid growth of Artificial Intelligence (AI) in healthcare has sparked interest in Trustworthy AI and AI Implementation Science, both of which are essential for accelerating clinical adoption. However, strict regulations, gaps between research and clinical settings, and challenges in evaluating AI systems continue to hinder real-world implementation. This study presents an AI implementation case study within Shriners Childrens (SC), a large multisite pediatric system, showcasing the modernization of SCs Research Data Warehouse (RDW) to OMOP CDM v5.4 within a secure Microsoft Fabric environment. We introduce a Python-based data quality assessment tool compatible with SCs infrastructure, extending OHDsi's R/Java-based Data Quality Dashboard (DQD) and integrating Trustworthy AI principles using the METRIC framework. This extension enhances data quality evaluation by addressing informative missingness, redundancy, timeliness, and distributional consistency. We also compare systematic and case-specific AI implementation strategies for Craniofacial Microsomia (CFM) using the FHIR standard. Our contributions include a real-world evaluation of AI implementations, integration of Trustworthy AI principles into data quality assessment, and insights into hybrid implementation strategies that blend systematic infrastructure with use-case-driven approaches to advance AI in healthcare.

Figures (5)

• Figure 1: Overview of our adaptation and implementation of OHDSI Data Quality Dashboard (DQD) within Shriners Children's MS Fabric environment.
• Figure 2: We modernized Shriners Children's Research Data Warehouse OMOP CDM database, and observed an improvement of the OHDSI DQD assessment, with more tests performed post-modernization.
• Figure 3: We calculated the completeness for each hospital site, and observed a slight difference in completeness.
• Figure 4: We represented the distribution of the different data for different data sources (hospital sites). We can observe that the distribution differs from one site to another.
• Figure 5: We show the AUROC for different models when using source codes (blue) vs. using harmonized OMOP CDM concept code (orange) vs. using supersets of OMOP CDM concept codes (green).