PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Support

TL;DR

PAIR-SAFE tackles the safety and clinical alignment gaps in AI-mediated mental health support by introducing a paired-agent framework where a Responder is supervised in real time by a MITI-4 grounded Judge. Using SeekerSim to model client behavior and MITI-based metrics to guide refinement, the study demonstrates significant improvements in key MI dimensions and relational quality over a Baseline setup. The approach provides transparent, auditable oversight without retraining the underlying language model, and yields simulated data and artifacts for benchmarking. Overall, the work highlights the potential of modular, supervisor-based oversight to enhance safety, accountability, and clinical alignment in AI-driven therapeutic conversations.

Abstract

Large language models (LLMs) are increasingly used for mental health support, yet they can produce responses that are overly directive, inconsistent, or clinically misaligned, particularly in sensitive or high-risk contexts. Existing approaches to mitigating these risks largely rely on implicit alignment through training or prompting, offering limited transparency and runtime accountability. We introduce PAIR-SAFE, a paired-agent framework for auditing and refining AI-generated mental health support that integrates a Responder agent with a supervisory Judge agent grounded in the clinically validated Motivational Interviewing Treatment Integrity (MITI-4) framework. The Judgeaudits each response and provides structuredALLOW or REVISE decisions that guide runtime response refinement. We simulate counseling interactions using a support-seeker simulator derived from human-annotated motivational interviewing data. We find that Judge-supervised interactions show significant improvements in key MITI dimensions, including Partnership, Seek Collaboration, and overall Relational quality. Our quantitative findings are supported by qualitative expert evaluation, which further highlights the nuances of runtime supervision. Together, our results reveal that such pairedagent approach can provide clinically grounded auditing and refinement for AI-assisted conversational mental health support.

Figures (2)

• Figure 1: A schematic overview of our study design with the Pair-Safe framework (detailed on the right).
• Figure 2: Cumulative pass rate of conversations meeting the quality threshold across revisions.