PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systems

TL;DR

PrivacyBench tackles the challenge of evaluating hybrid privacy configurations in vision systems by introducing a four-layer, YAML-driven benchmarking framework with integrated energy monitoring and deterministic execution. It systematically analyzes interactions among Federated Learning (FL), Differential Privacy (DP), and Secure Multi-Party Computation (SMPC) on CNN and transformer architectures (ResNet18 and ViT) using privacy-sensitive medical datasets (Alzheimer MRI and ISIC). The key finding is non-additive behavior: FL+SMPC largely preserves utility with modest overhead, whereas FL+DP can cause severe convergence failure and substantial energy and time costs, varying with architecture. These results highlight the need for co-design and principled system-level evaluation to ensure privacy-preserving vision deployments are feasible in practice and resource-constrained settings.

Abstract

Privacy preserving machine learning deployments in sensitive deep learning applications; from medical imaging to autonomous systems; increasingly require combining multiple techniques. Yet, practitioners lack systematic guidance to assess the synergistic and non-additive interactions of these hybrid configurations, relying instead on isolated technique analysis that misses critical system level interactions. We introduce PrivacyBench, a benchmarking framework that reveals striking failures in privacy technique combinations with severe deployment implications. Through systematic evaluation across ResNet18 and ViT models on medical datasets, we uncover that FL + DP combinations exhibit severe convergence failure, with accuracy dropping from 98% to 13% while compute costs and energy consumption substantially increase. In contrast, FL + SMPC maintains near-baseline performance with modest overhead. Our framework provides the first systematic platform for evaluating privacy-utility-cost trade-offs through automated YAML configuration, resource monitoring, and reproducible experimental protocols. PrivacyBench enables practitioners to identify problematic technique interactions before deployment, moving privacy-preserving computer vision from ad-hoc evaluation toward principled systems design. These findings demonstrate that privacy techniques cannot be composed arbitrarily and provide critical guidance for robust deployment in resource-constrained environments.

Figures (2)

• Figure 1: PrivacyBench Architecture Overview. A four-layer modular framework enabling systematic evaluation of privacy technique interactions. The Configuration Layer handles YAML-based experiment specification, the Modular Layer supports diverse privacy combinations, the Execution Layer integrates comprehensive resource monitoring, and the Output Layer generates reproducible results. This architecture enables controlled evaluation of hybrid privacy configurations while tracking computational costs and energy consumption—capabilities missing from existing evaluation frameworks.
• Figure 2: Systematic analysis of privacy-utility-cost trade-offs. The figure shows that FL and FL+SMPC offer superior trade-offs, achieving near-baseline accuracy with minimal overhead. In contrast, all FL+DP variants result in significant accuracy degradation and variable computational costs, highlighting their unsuitability for practical deployment. The "preferred region" in the lower-left corner indicates desirable combinations of low cost and high utility.