Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models

Daniel Lopez-Martinez

TL;DR

The paper addresses the risk that generalist GenAI systems can promote unapproved medical product uses. It proposes a four-stage detection pipeline—input standardization, NER for drugs and indications, FDA-label mapping via FDALabel embeddings, and off-label identification using a T5-large zero-shot check—to identify off-label promotion in multimodal outputs, demonstrated on Claude 3. A synthetic red-teaming approach yielded 14,300 queries across 35 drugs, revealing off-label prompts in 15.4% of responses and achieving precision 85.75%, recall 80.47%, and F1 83.02% on a 2,000-sample human-annotated subset. The work underscores the need for guardrails to ensure FDA-approved labeling adherence in GenAI medical applications and offers a concrete, scalable method for post-hoc monitoring, while noting limitations such as scope and model generalizability.

Abstract

Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.

Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models

TL;DR

The paper addresses the risk that generalist GenAI systems can promote unapproved medical product uses. It proposes a four-stage detection pipeline—input standardization, NER for drugs and indications, FDA-label mapping via FDALabel embeddings, and off-label identification using a T5-large zero-shot check—to identify off-label promotion in multimodal outputs, demonstrated on Claude 3. A synthetic red-teaming approach yielded 14,300 queries across 35 drugs, revealing off-label prompts in 15.4% of responses and achieving precision 85.75%, recall 80.47%, and F1 83.02% on a 2,000-sample human-annotated subset. The work underscores the need for guardrails to ensure FDA-approved labeling adherence in GenAI medical applications and offers a concrete, scalable method for post-hoc monitoring, while noting limitations such as scope and model generalizability.

Abstract

Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.
Paper Structure (15 sections, 2 figures, 1 table)

This paper contains 15 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Regulatory framework
  4. Harms of off-label promotion by GenAI
  5. Detecting off-label use with ML
  6. Methods
  7. Input standardization
  8. Named entity recognition
  9. Product and indication recognition
  10. Off-label identification
  11. Experimental Setup
  12. Model
  13. Synthetic user query generation
  14. Experimental Results
  15. Conclusion

Figures (2)

  • Figure 1: Overview of a generative artificial intelligent system for medical product question answering, product recommendation, and general medical question answering. Such system may handle a handle a diverse range of biomedical data modalities, and use a number of evidence sources, some of which are not approved by the FDA.
  • Figure 2: Process flow for GenAI model generation (top) and evaluation of the responses for off-label identification (bottom).