Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports

Henning Schäfer, Cynthia S. Schmidt, Johannes Wutzkowsky, Kamil Lorek, Lea Reinartz, Johannes Rückert, Christian Temme, Britta Böckmann, Peter A. Horn, Christoph M. Friedrich

TL;DR

The paper addresses the persistence of paper-based transfusion reaction reporting by introducing an open-source multimodal pipeline that automates checkbox data extraction from scanned forms. It combines a YOLOv8-based checkbox detector with two extraction pathways—OCR plus Levenshtein matching and a vision-language model prompting approach— and validates performance against gold-standard annual reports from 2017 to 2024. Results show that the VLM-based method achieves higher precision, recall, and overall category-mapping accuracy than the OCR-based baseline, with barcode detection also yielding high accuracies; the pipeline significantly reduces manual data entry while maintaining reliability. The approach is multilingual, adaptable to other checkbox-rich documents, and designed for self-hosted deployment, offering a scalable, open-source framework for improving regulatory reporting and data-driven decision-making in healthcare.

Abstract

Despite the growing adoption of electronic health records, many processes still rely on paper documents, reflecting the heterogeneous real-world conditions in which healthcare is delivered. The manual transcription process is time-consuming and prone to errors when transferring paper-based data to digital formats. To streamline this workflow, this study presents an open-source pipeline that extracts and categorizes checkbox data from scanned documents. Demonstrated on transfusion reaction reports, the design supports adaptation to other checkbox-rich document types. The proposed method integrates checkbox detection, multilingual optical character recognition (OCR) and multilingual vision-language models (VLMs). The pipeline achieves high precision and recall compared against annually compiled gold-standards from 2017 to 2024. The result is a reduction in administrative workload and accurate regulatory reporting. The open-source availability of this pipeline encourages self-hosted parsing of checkbox forms.

A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports

TL;DR

The paper addresses the persistence of paper-based transfusion reaction reporting by introducing an open-source multimodal pipeline that automates checkbox data extraction from scanned forms. It combines a YOLOv8-based checkbox detector with two extraction pathways—OCR plus Levenshtein matching and a vision-language model prompting approach— and validates performance against gold-standard annual reports from 2017 to 2024. Results show that the VLM-based method achieves higher precision, recall, and overall category-mapping accuracy than the OCR-based baseline, with barcode detection also yielding high accuracies; the pipeline significantly reduces manual data entry while maintaining reliability. The approach is multilingual, adaptable to other checkbox-rich documents, and designed for self-hosted deployment, offering a scalable, open-source framework for improving regulatory reporting and data-driven decision-making in healthcare.

Abstract

Despite the growing adoption of electronic health records, many processes still rely on paper documents, reflecting the heterogeneous real-world conditions in which healthcare is delivered. The manual transcription process is time-consuming and prone to errors when transferring paper-based data to digital formats. To streamline this workflow, this study presents an open-source pipeline that extracts and categorizes checkbox data from scanned documents. Demonstrated on transfusion reaction reports, the design supports adaptation to other checkbox-rich document types. The proposed method integrates checkbox detection, multilingual optical character recognition (OCR) and multilingual vision-language models (VLMs). The pipeline achieves high precision and recall compared against annually compiled gold-standards from 2017 to 2024. The result is a reduction in administrative workload and accurate regulatory reporting. The open-source availability of this pipeline encourages self-hosted parsing of checkbox forms.
Paper Structure (18 sections, 1 figure, 4 tables)

This paper contains 18 sections, 1 figure, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. Contributions
  3. RELATED WORK
  4. Vision-Language Models for Document Understanding
  5. Checkbox Detection and Form Parsing
  6. DATA
  7. Data Source and Characteristics
  8. Ethical and Regulatory Considerations
  9. METHODS
  10. Barcode Detection
  11. Approach 1: OCR-Based Extraction with Levenshtein Matching
  12. Approach 2: Vision-Language Model (VLM)-Based Prompting
  13. RESULTS
  14. Barcode Detection
  15. Findings and Suspected Diagnosis Category Mapping
  16. ...and 3 more sections

Figures (1)

  • Figure 1: Illustration of the YOLO-based checkbox detection approach applied to a complete transfusion reaction report. Each blue bounding box denotes a detected checkbox area. The model is used to extract the relevant bounding boxes of a contiguous checkbox area to be fed into the Vision Language Model.