Revisiting the MIMIC-IV Benchmark: Experiments Using Language Models for Electronic Health Records
Jesus Lovon, Thouria Ben-Haddi, Jules Di Scala, Jose G. Moreno, Lynda Tamine
TL;DR
This work revisits the MIMIC-IV benchmark to address the lack of standardized, text-focused evaluation for EHR tasks. It integrates MIMIC-IV into the Hugging Face datasets ecosystem and introduces data-to-text templates to convert tabular ICU data into textual inputs, enabling direct evaluation with both transformer models and LLMs. Across mortality prediction tasks, fine-tuned text-based models show performance competitive with strong tabular baselines, while zero-shot LLMs struggle to leverage EHR representations, highlighting the potential and current limits of text-centric approaches in clinical settings. The study provides a reproducible framework and actionable insights to guide future benchmarking and model development in health NLP.
Abstract
The lack of standardized evaluation benchmarks in the medical domain for text inputs can be a barrier to widely adopting and leveraging the potential of natural language models for health-related downstream tasks. This paper revisited an openly available MIMIC-IV benchmark for electronic health records (EHRs) to address this issue. First, we integrate the MIMIC-IV data within the Hugging Face datasets library to allow an easy share and use of this collection. Second, we investigate the application of templates to convert EHR tabular data to text. Experiments using fine-tuned and zero-shot LLMs on the mortality of patients task show that fine-tuned text-based models are competitive against robust tabular classifiers. In contrast, zero-shot LLMs struggle to leverage EHR representations. This study underlines the potential of text-based approaches in the medical field and highlights areas for further improvement.