Towards Automatic Evaluation and Selection of PHI De-identification Models via Multi-Agent Collaboration

TL;DR

TEAM-PHI presents a multi-agent framework that uses several LLM-based Evaluation Agents to automatically evaluate PHI de-identification outputs and select the best model without heavy reliance on gold labels. By decoupling PHI extraction from evaluation and aggregating judgments with LLM-based majority voting, the approach yields stable, reproducible rankings that align with ground-truth and human assessments on real-world clinical notes. Experiments show Llama-70B and GPT-4o consistently rank highly, with Llama-70B providing the most reliable performance across PHI categories, especially PERSON. The framework offers a practical, secure, and scalable pathway for benchmarking and deploying privacy-preserving de-identification in healthcare settings, while outlining limitations and directions for extending coverage and calibration.

Abstract

Protected health information (PHI) de-identification is critical for enabling the safe reuse of clinical notes, yet evaluating and comparing PHI de-identification models typically depends on costly, small-scale expert annotations. We present TEAM-PHI, a multi-agent evaluation and selection framework that uses large language models (LLMs) to automatically measure de-identification quality and select the best-performing model without heavy reliance on gold labels. TEAM-PHI deploys multiple Evaluation Agents, each independently judging the correctness of PHI extractions and outputting structured metrics. Their results are then consolidated through an LLM-based majority voting mechanism that integrates diverse evaluator perspectives into a single, stable, and reproducible ranking. Experiments on a real-world clinical note corpus demonstrate that TEAM-PHI produces consistent and accurate rankings: despite variation across individual evaluators, LLM-based voting reliably converges on the same top-performing systems. Further comparison with ground-truth annotations and human evaluation confirms that the framework's automated rankings closely match supervised evaluation. By combining independent evaluation agents with LLM majority voting, TEAM-PHI offers a practical, secure, and cost-effective solution for automatic evaluation and best-model selection in PHI de-identification, even when ground-truth labels are limited.

Figures (1)

• Figure 1: Framework of TEAM-PHI.