Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I-SIRch: AI-Powered Concept Annotation Tool For Equitable Extraction And Analysis Of Safety Insights From Maternity Investigations

Mohit Kumar Singh, Georgina Cosma, Patrick Waterson, Jonathan Back, Gyuchan Thomas Jun

TL;DR

The paper introduces I-SIRch, an AI-assisted framework that automates the annotation of maternity incident reports with human-factor concepts from the SIRch taxonomy, addressing the gap in socio-technical safety analysis. It combines text preprocessing, extraction, selective sentence tagging, manual annotation, active learning, and human verification in a human-in-the-loop pipeline to produce a trained model capable of applying multiple SIRch concepts to sentences. Evaluations across synthetic and real data show high performance, with balanced accuracy around $0.90$ on real data after retraining, and wide coverage of human factors across ethnic groups, highlighting both potential and biases. The work demonstrates how socio-technical insights can be scaled in healthcare safety analysis, enabling targeted interventions to reduce disparities and improve maternal outcomes by integrating human factors with biomedical analysis.

Abstract

Maternity care is a complex system involving treatments and interactions between patients, providers, and the care environment. To improve patient safety and outcomes, understanding the human factors (e.g. individuals decisions, local facilities) influencing healthcare delivery is crucial. However, most current tools for analysing healthcare data focus only on biomedical concepts (e.g. health conditions, procedures and tests), overlooking the importance of human factors. We developed a new approach called I-SIRch, using artificial intelligence to automatically identify and label human factors concepts in maternity healthcare investigation reports describing adverse maternity incidents produced by England's Healthcare Safety Investigation Branch (HSIB). These incident investigation reports aim to identify opportunities for learning and improving maternal safety across the entire healthcare system. I-SIRch was trained using real data and tested on both real and simulated data to evaluate its performance in identifying human factors concepts. When applied to real reports, the model achieved a high level of accuracy, correctly identifying relevant concepts in 90\% of the sentences from 97 reports. Applying I-SIRch to analyse these reports revealed that certain human factors disproportionately affected mothers from different ethnic groups. Our work demonstrates the potential of using automated tools to identify human factors concepts in maternity incident investigation reports, rather than focusing solely on biomedical concepts. This approach opens up new possibilities for understanding the complex interplay between social, technical, and organisational factors influencing maternal safety and population health outcomes. By taking a more comprehensive view of maternal healthcare delivery, we can develop targeted interventions to address disparities and improve maternal outcomes.

I-SIRch: AI-Powered Concept Annotation Tool For Equitable Extraction And Analysis Of Safety Insights From Maternity Investigations

TL;DR

The paper introduces I-SIRch, an AI-assisted framework that automates the annotation of maternity incident reports with human-factor concepts from the SIRch taxonomy, addressing the gap in socio-technical safety analysis. It combines text preprocessing, extraction, selective sentence tagging, manual annotation, active learning, and human verification in a human-in-the-loop pipeline to produce a trained model capable of applying multiple SIRch concepts to sentences. Evaluations across synthetic and real data show high performance, with balanced accuracy around on real data after retraining, and wide coverage of human factors across ethnic groups, highlighting both potential and biases. The work demonstrates how socio-technical insights can be scaled in healthcare safety analysis, enabling targeted interventions to reduce disparities and improve maternal outcomes by integrating human factors with biomedical analysis.

Abstract

Maternity care is a complex system involving treatments and interactions between patients, providers, and the care environment. To improve patient safety and outcomes, understanding the human factors (e.g. individuals decisions, local facilities) influencing healthcare delivery is crucial. However, most current tools for analysing healthcare data focus only on biomedical concepts (e.g. health conditions, procedures and tests), overlooking the importance of human factors. We developed a new approach called I-SIRch, using artificial intelligence to automatically identify and label human factors concepts in maternity healthcare investigation reports describing adverse maternity incidents produced by England's Healthcare Safety Investigation Branch (HSIB). These incident investigation reports aim to identify opportunities for learning and improving maternal safety across the entire healthcare system. I-SIRch was trained using real data and tested on both real and simulated data to evaluate its performance in identifying human factors concepts. When applied to real reports, the model achieved a high level of accuracy, correctly identifying relevant concepts in 90\% of the sentences from 97 reports. Applying I-SIRch to analyse these reports revealed that certain human factors disproportionately affected mothers from different ethnic groups. Our work demonstrates the potential of using automated tools to identify human factors concepts in maternity incident investigation reports, rather than focusing solely on biomedical concepts. This approach opens up new possibilities for understanding the complex interplay between social, technical, and organisational factors influencing maternal safety and population health outcomes. By taking a more comprehensive view of maternal healthcare delivery, we can develop targeted interventions to address disparities and improve maternal outcomes.
Paper Structure (29 sections, 6 equations, 7 figures, 4 tables)

This paper contains 29 sections, 6 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Reports.
  3. New reports.
  4. SIRch taxonomy.
  5. Text preprocessing module.
  6. Text extraction module.
  7. Text selection module.
  8. Manual annotation of selected text.
  9. Machine learning module.
  10. Human verification.
  11. Performance monitoring.
  12. Inter-rater reliability (IRR).
  13. True Positive (TP).
  14. False Positive (FP).
  15. False Negative (FN).
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Proposed I-SIRch framework
  • Figure 2: Text selection module
  • Figure 3: Total number of concepts and average number of concepts per report across ethnic groups.
  • Figure 4: Percentage of mothers and families from each ethnic group in our dataset compared to England's 2021 census data.
  • Figure 5: Average number of correct and incorrect concepts across various ethnic groups (Batch 1).
  • ...and 2 more figures