Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops

Zainab Ghafoor, Md Shafiqul Islam, Koushik Howlader, Md Rasel Khondokar, Tanusree Bhattacharjee, Sayantan Chakraborty, Adrito Roy, Ushashi Bhattacharjee, Tirtho Roy

TL;DR

This work tackles the safety and ethical risks of medical LLMs by introducing an inference-time, non-retraining multi-agent refinement loop. It pairs two generators (DeepSeek R1 and Med-PaLM) with two evaluators (LLaMA 3.1 for AMA ethics and Phi-4 for SRA risk) to iteratively revise outputs across 900 adversarial prompts, guided by AMA ethics and SRA-5 criteria. The results show substantial safety gains, with an 89% reduction in ethics violations and a 92% reduction in risk levels, achieved within five iterations for the vast majority of prompts. Importantly, the approach avoids retraining the generator, enabling regulator-aligned, adaptable safety governance that can be updated by altering evaluation policies and prompts. The study demonstrates a scalable pathway to safer medical AI, while acknowledging the need for clinician calibration and broader validation across models and clinical contexts.

Abstract

Large Language Models (LLMs) are increasingly applied in healthcare, yet ensuring their ethical integrity and safety compliance remains a major barrier to clinical deployment. This work introduces a multi-agent refinement framework designed to enhance the safety and reliability of medical LLMs through structured, iterative alignment. Our system combines two generative models - DeepSeek R1 and Med-PaLM - with two evaluation agents, LLaMA 3.1 and Phi-4, which assess responses using the American Medical Association's (AMA) Principles of Medical Ethics and a five-tier Safety Risk Assessment (SRA-5) protocol. We evaluate performance across 900 clinically diverse queries spanning nine ethical domains, measuring convergence efficiency, ethical violation reduction, and domain-specific risk behavior. Results demonstrate that DeepSeek R1 achieves faster convergence (mean 2.34 vs. 2.67 iterations), while Med-PaLM shows superior handling of privacy-sensitive scenarios. The iterative multi-agent loop achieved an 89% reduction in ethical violations and a 92% risk downgrade rate, underscoring the effectiveness of our approach. This study presents a scalable, regulator-aligned, and cost-efficient paradigm for governing medical AI safety.

Improving the Safety and Trustworthiness of Medical AI via Multi-Agent Evaluation Loops

TL;DR

This work tackles the safety and ethical risks of medical LLMs by introducing an inference-time, non-retraining multi-agent refinement loop. It pairs two generators (DeepSeek R1 and Med-PaLM) with two evaluators (LLaMA 3.1 for AMA ethics and Phi-4 for SRA risk) to iteratively revise outputs across 900 adversarial prompts, guided by AMA ethics and SRA-5 criteria. The results show substantial safety gains, with an 89% reduction in ethics violations and a 92% reduction in risk levels, achieved within five iterations for the vast majority of prompts. Importantly, the approach avoids retraining the generator, enabling regulator-aligned, adaptable safety governance that can be updated by altering evaluation policies and prompts. The study demonstrates a scalable pathway to safer medical AI, while acknowledging the need for clinician calibration and broader validation across models and clinical contexts.

Abstract

Large Language Models (LLMs) are increasingly applied in healthcare, yet ensuring their ethical integrity and safety compliance remains a major barrier to clinical deployment. This work introduces a multi-agent refinement framework designed to enhance the safety and reliability of medical LLMs through structured, iterative alignment. Our system combines two generative models - DeepSeek R1 and Med-PaLM - with two evaluation agents, LLaMA 3.1 and Phi-4, which assess responses using the American Medical Association's (AMA) Principles of Medical Ethics and a five-tier Safety Risk Assessment (SRA-5) protocol. We evaluate performance across 900 clinically diverse queries spanning nine ethical domains, measuring convergence efficiency, ethical violation reduction, and domain-specific risk behavior. Results demonstrate that DeepSeek R1 achieves faster convergence (mean 2.34 vs. 2.67 iterations), while Med-PaLM shows superior handling of privacy-sensitive scenarios. The iterative multi-agent loop achieved an 89% reduction in ethical violations and a 92% risk downgrade rate, underscoring the effectiveness of our approach. This study presents a scalable, regulator-aligned, and cost-efficient paradigm for governing medical AI safety.
Paper Structure (29 sections, 1 figure, 5 tables, 1 algorithm)

This paper contains 29 sections, 1 figure, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Language Models
  4. LLM Safety and Alignment
  5. Multi Agent Systems
  6. Methodology
  7. System Overview and Agent Roles
  8. Generative models.
  9. Evaluator agents.
  10. Safety Evaluation Criteria
  11. AMA 9 Ethics Scoring
  12. SRA 5 Risk Levels
  13. Iterative Refinement Loop
  14. Stopping Rule and Iteration Budget
  15. Consensus and Feedback Construction
  16. ...and 14 more sections

Figures (1)

  • Figure 1: Multi agent evaluation loop. A generator produces an initial response. Two evaluators assess the response in parallel using ethics and risk criteria. Their outputs are merged into a single consensus decision. If the response meets the safety thresholds, the loop stops. Otherwise, structured feedback is generated and the generator revises its answer.