Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transparency-First Medical Language Models: Datasheets, Model Cards, and End-to-End Data Provenance for Clinical NLP

Olaf Yunus Laitinen Imanov, Taner Yilmaz, Ayse Tuba Tugrul, Melike Nesrin Zaman, Ozkan Gunalp, Duygu Erisken, Sila Burde Dulger, Rana Irem Turhan, Izzet Ozdemir, Derya Umut Kulali, Ozan Akbulut, Harun Demircioglu, Hasan Basri Kara, Berfin Tavan

TL;DR

The paper addresses the need for auditable, privacy-conscious reporting in clinical NLP by introducing TeMLM, a transparency-first framework. It defines a trio of artifacts—TeMLM-Datasheet, TeMLM-Card, and TeMLM-Provenance—plus a lightweight conformance checklist to enable machine-checkable governance. A worked example on Technetium-I with ProtactiniumBERT demonstrates how to document data provenance, de-identification, and evaluation for PHI de-identification and ICD coding tasks, while acknowledging the value of synthetic benchmarks for tooling and process validation. The framework aims to raise reporting standards, support reproducibility across restricted data environments, and facilitate safer deployment through principled transparency, provenance, and monitoring.

Abstract

We introduce TeMLM, a set of transparency-first release artifacts for clinical language models. TeMLM unifies provenance, data transparency, modeling transparency, and governance into a single, machine-checkable release bundle. We define an artifact suite (TeMLM-Card, TeMLM-Datasheet, TeMLM-Provenance) and a lightweight conformance checklist for repeatable auditing. We instantiate the artifacts on Technetium-I, a large-scale synthetic clinical NLP dataset with 498,000 notes, 7.74M PHI entity annotations across 10 types, and ICD-9-CM diagnosis labels, and report reference results for ProtactiniumBERT (about 100 million parameters) on PHI de-identification (token classification) and top-50 ICD-9 code extraction (multi-label classification). We emphasize that synthetic benchmarks are valuable for tooling and process validation, but models should be validated on real clinical data prior to deployment.

Transparency-First Medical Language Models: Datasheets, Model Cards, and End-to-End Data Provenance for Clinical NLP

TL;DR

The paper addresses the need for auditable, privacy-conscious reporting in clinical NLP by introducing TeMLM, a transparency-first framework. It defines a trio of artifacts—TeMLM-Datasheet, TeMLM-Card, and TeMLM-Provenance—plus a lightweight conformance checklist to enable machine-checkable governance. A worked example on Technetium-I with ProtactiniumBERT demonstrates how to document data provenance, de-identification, and evaluation for PHI de-identification and ICD coding tasks, while acknowledging the value of synthetic benchmarks for tooling and process validation. The framework aims to raise reporting standards, support reproducibility across restricted data environments, and facilitate safer deployment through principled transparency, provenance, and monitoring.

Abstract

We introduce TeMLM, a set of transparency-first release artifacts for clinical language models. TeMLM unifies provenance, data transparency, modeling transparency, and governance into a single, machine-checkable release bundle. We define an artifact suite (TeMLM-Card, TeMLM-Datasheet, TeMLM-Provenance) and a lightweight conformance checklist for repeatable auditing. We instantiate the artifacts on Technetium-I, a large-scale synthetic clinical NLP dataset with 498,000 notes, 7.74M PHI entity annotations across 10 types, and ICD-9-CM diagnosis labels, and report reference results for ProtactiniumBERT (about 100 million parameters) on PHI de-identification (token classification) and top-50 ICD-9 code extraction (multi-label classification). We emphasize that synthetic benchmarks are valuable for tooling and process validation, but models should be validated on real clinical data prior to deployment.
Paper Structure (60 sections, 2 equations, 9 figures, 15 tables)

This paper contains 60 sections, 2 equations, 9 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Documentation standards: datasheets and model cards
  4. Clinical NLP datasets and privacy constraints
  5. Provenance and reproducibility
  6. Reporting guidance for clinical AI
  7. Transparency as a reproducibility primitive
  8. From checklists to machine-readability.
  9. Distribution shift, contamination, and leakage
  10. Why this matters for medical LMs.
  11. TeMLM transparency pillar: scope and principles
  12. Principles
  13. Methods: TeMLM artifacts
  14. TeMLM-Datasheet (dataset documentation)
  15. Documentation levels and uncertainty.
  16. ...and 45 more sections

Figures (9)

  • Figure 1: Illustrative completeness profile for mandatory TeMLM-Datasheet sections. Completeness can be tracked over dataset versions to detect documentation drift and missing disclosures.
  • Figure 2: Technetium-I corpus composition by note type for the worked example (498,000 notes). TeMLM-Datasheet recommends stratifying key statistics by clinically meaningful slices such as note type and care setting.
  • Figure 3: Leakage audit curve for the worked example: the fraction of test notes with near-duplicate similarity above threshold (lower is better). Reporting the full curve avoids cherry-picking a single threshold and supports reproducible audits.
  • Figure 4: Illustrative documentation-drift trace across dataset versions. TeMLM treats documentation as a versioned artifact: completeness should be monitored, and regressions should block release until disclosures are restored.
  • Figure 5: Illustrative PHI de-identification performance (micro-F1 on Technetium-I test split).
  • ...and 4 more figures