MIBench: A Comprehensive Framework for Benchmarking Model Inversion Attack and Defense

TL;DR

Model inversion attacks threaten privacy by reconstructing training data from model outputs. The paper introduces MIBench, a comprehensive benchmark with a modular toolbox that integrates 19 MI attack/defense methods and 9 evaluation protocols to enable reproducible, large-scale comparisons. Through extensive experiments across resolutions, model predictive power, defense strategies, and adversarial robustness, the authors reveal how attack effectiveness scales with target accuracy and how defenses perform under diverse conditions. The benchmark provides a foundation for fair evaluation and guides future research toward robust privacy-preserving MI defenses.

Abstract

Model Inversion (MI) attacks aim at leveraging the output information of target models to reconstruct privacy-sensitive training data, raising critical concerns regarding the privacy vulnerabilities of Deep Neural Networks (DNNs). Unfortunately, in tandem with the rapid evolution of MI attacks, the absence of a comprehensive benchmark with standardized metrics and reproducible implementations has emerged as a formidable challenge. This deficiency has hindered objective comparison of methodological advancements and reliable assessment of defense efficacy. To address this critical gap, we build the first practical benchmark named MIBench for systematic evaluation of model inversion attacks and defenses. This benchmark bases on an extensible and reproducible modular-based toolbox which currently integrates a total of 19 state-of-the-art attack and defense methods and encompasses 9 standardized evaluation protocols. Capitalizing on this foundation, we conduct extensive evaluation from multiple perspectives to holistically compare and analyze various methods across different scenarios, such as the impact of target resolution, model predictive power, defense performance and adversarial robustness.

Figures (10)

• Figure 1: Overview of the basic structure of modular-based toolbox for our benchmark.
• Figure 2: Visual comparison between different MI attacks. As MI attacks are designed to reconstruct private features of target images at the distribution level rather than strictly full reconstruction of original images, it is sufficient for the inverted images to exhibit discriminative characteristics associated with the original image category.
• Figure 3: Comparison across ResNet-152 with varied predictive power. (a) The incremental trend of $Acc@1$ metric on different attack methods. (b) The decreasing trend of $\delta_{face}$ metric on different attack methods.
• Figure 4: Evaluation of Different white-box MI attacks on multiple MI defense strategies. Notably, attackers can obtain full access to the model parameters under the white-box settings, enabling them to directly circumvent inference-time defense plugins. Consequently, such defenses are invalid to white-box attacks.
• Figure 5: Evaluation of Different black-box and label-only MI attacks on multiple MI defense strategies.