Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology

TL;DR

The paper addresses the challenge of translating AI for nasogastric tube (NGT) placement verification on Chest X-rays into routine ICU practice. It adopts a human-centered, context-rich case study, using contextual inquiry and interviews with 15 clinical stakeholders to illuminate end-to-end workflows, data practices, and safeguarding mechanisms. A key contribution is a systematic mapping framework and the concept of Human-Process Integration of AI, which embeds AI within existing safeguarding and information-review processes to balance patient safety, workflow efficiency, and clinician trust. The findings highlight complex interdependencies among people, technology, and organizational factors, and offer design guidance that can inform clinically useful, ethical AI deployment in radiology and potentially other critical care tasks with similar workflow dynamics.

Abstract

Nasogastric tubes (NGTs) are feeding tubes that are inserted through the nose into the stomach to deliver nutrition or medication. If not placed correctly, they can cause serious harm, even death to patients. Recent AI developments demonstrate the feasibility of robustly detecting NGT placement from Chest X-ray images to reduce risks of sub-optimally or critically placed NGTs being missed or delayed in their detection, but gaps remain in clinical practice integration. In this study, we present a human-centered approach to the problem and describe insights derived following contextual inquiry and in-depth interviews with 15 clinical stakeholders. The interviews helped understand challenges in existing workflows, and how best to align technical capabilities with user needs and expectations. We discovered the trade-offs and complexities that need consideration when choosing suitable workflow stages, target users, and design configurations for different AI proposals. We explored how to balance AI benefits and risks for healthcare staff and patients within broader organizational and medical-legal constraints. We also identified data issues related to edge cases and data biases that affect model training and evaluation; how data documentation practices influence data preparation and labelling; and how to measure relevant AI outcomes reliably in future evaluations. We discuss how our work informs design and development of AI applications that are clinically useful, ethical, and acceptable in real-world healthcare services.

Figures (3)

• Figure 1: Overview of ICU end-to-end workflow of CXR based NGT verification from initial insertion through to decisions to use NGT for feeding or correct it, including temporal delays. The red exclamation mark symbol illustrates areas of potential integration of AI assisted critical NGT misplacement detection at different time-points and for different user groups. Note that this workflow differs from Stroke care where clinicians are not present at CXR capture, nor often review the image themselves, and rely on official radiology reporting. In ICU, NGTs are most commonly already acted upon by the time of the official radiology report.
• Figure 2: Schematic representation of the '.NGT' template used to verify NGT placement outside an official radiology report.
• Figure 3: Left: Simplified illustration of a patient’s key anatomical structures of relevance to NGT placement assessment (the heart is drawn for clarity only). Right: different NGT placements: A) correct placement of NGT with tip projecting in the stomach; B) critical misplacement of the NGT into the patient's right lung (via the trachea and the right main bronchus); C) misplaced NGT which is coiled in patient’s mid oesophagus; and D) misplaced NGT with tip in distal oesophagus requiring further advancement to reach the stomach.