Balancing Safety and Helpfulness in Healthcare AI Assistants through Iterative Preference Alignment

TL;DR

This study introduces an iterative post-deployment safety-alignment framework that blends Kahneman-Tversky Optimization (KTO) and Direct Preference Optimization (DPO) to tune healthcare-oriented LLMs against domain-specific safety signals. Using CARES-18K, the authors benchmark four open-source models and demonstrate substantial safety gains (up to 42% in safety scores) while revealing architecture-dependent trade-offs in helpfulness and refusal behavior. The work compares self-evaluation against external judgments, highlighting calibration biases and the context-dependent reliability of self-assessment, and it analyzes how validator choice influences policy selection (KTO vs. DPO) across training cycles. The results advocate a modular, post-deployment safety toolkit, emphasize human-in-the-loop oversight, and discuss practical deployment considerations under evolving healthcare standards. Overall, the paper provides a practical, architecture-aware approach to improving safety and clinical usefulness in medical AI assistants through iterative preference alignment.

Abstract

Large Language Models (LLMs) are increasingly used in healthcare, yet ensuring their safety and trustworthiness remains a barrier to deployment. Conversational medical assistants must avoid unsafe compliance without over-refusing benign queries. We present an iterative post-deployment alignment framework that applies Kahneman-Tversky Optimization (KTO) and Direct Preference Optimization (DPO) to refine models against domain-specific safety signals. Using the CARES-18K benchmark for adversarial robustness, we evaluate four LLMs (Llama-3B/8B, Meditron-8B, Mistral-7B) across multiple cycles. Our results show up to 42% improvement in safety-related metrics for harmful query detection, alongside interesting trade-offs against erroneous refusals, thereby exposing architecture-dependent calibration biases. We also perform ablation studies to identify when self-evaluation is reliable and when external or finetuned judges are necessary to maximize performance gains. Our findings underscore the importance of adopting best practices that balance patient safety, user trust, and clinical utility in the design of conversational medical assistants.

Figures (21)

• Figure 1: Example prompts in CARES dataset at 4 harmful levels.
• Figure 2: Results on test set after 1 cycle of training using self-evaluation regimen.
• Figure 3: Progression of evaluation metrics across 5 iterations, using self-evaluation for the target LLMs on the validation set. Solid lines represents metrics based on safety judgment by GPT-4o-mini. Dashed lines correspond to using the target model's self-generated Safety Judgments. Squares mark KTO results and circles mark DPO results. Within each cycle, the higher-scoring method is shown with a filled marker; the lower one remains hollow.
• Figure 4: Progression of metrics across all 5 iterations using finetuned Llama-3B as the source of safety judgment for Meditron-Ext and Mistral-Ext on the validation set. Solid lines represent metrics based on GPT-4o-mini's safety judgement. Dashed lines represent the counterpart using Llama-3B's judgment. Squares mark KTO results and circles mark DPO results. Within each cycle, the higher-scoring method is shown with a filled marker; the lower one remains hollow. Metrics by non-finetuned judge tend to converge/overlap with GPT better than otherwise.
• Figure 5: SS and ERR on the test set of the model variants with the best metrics on the validation set for each experimental regimen. Base: baseline non-finetuned version; SELF: using self-evaluation; EXT: using finetuned Llama-3B as external judge; LMA: using non-finetuned Llama-3B as judge.