Federated Computation of ROC and PR Curves
Xuefeng Xu, Graham Cormode
TL;DR
The paper tackles the challenge of evaluating binary classifiers in federated learning without sharing raw scores by proposing a quantile-based, privacy-preserving method to approximate ROC and PR curves. It combines histogram-based quantile estimation with distributed differential privacy and monotone interpolation to reconstruct curves from quantiles, achieving provable area-error bounds of $O(1/Q)$ for ROC and $\tilde{O}(1/Q)$ for PR (with data-imbalance dependent refinements) while keeping communication linear in $Q$. The approach demonstrates strong empirical performance on real datasets under privacy constraints, with favorable comparisons to prior range-query methods and robust behavior under varying class distributions and score models. The method extends to multi-class settings and other metrics, offering a practical, scalable solution for privacy-preserving model evaluation in federated systems and enabling more reliable deployment decisions in privacy-sensitive domains.
Abstract
Receiver Operating Characteristic (ROC) and Precision-Recall (PR) curves are fundamental tools for evaluating machine learning classifiers, offering detailed insights into the trade-offs between true positive rate vs. false positive rate (ROC) or precision vs. recall (PR). However, in Federated Learning (FL) scenarios, where data is distributed across multiple clients, computing these curves is challenging due to privacy and communication constraints. Specifically, the server cannot access raw prediction scores and class labels, which are used to compute the ROC and PR curves in a centralized setting. In this paper, we propose a novel method for approximating ROC and PR curves in a federated setting by estimating quantiles of the prediction score distribution under distributed differential privacy. We provide theoretical bounds on the Area Error (AE) between the true and estimated curves, demonstrating the trade-offs between approximation accuracy, privacy, and communication cost. Empirical results on real-world datasets demonstrate that our method achieves high approximation accuracy with minimal communication and strong privacy guarantees, making it practical for privacy-preserving model evaluation in federated systems.