Toward Efficient Inference Attacks: Shadow Model Sharing via Mixture-of-Experts

TL;DR

This paper addresses the high computational cost of shadow-model-based inference attacks by introducing SHAPOOL, an MoE-based shadow pool that trains shared sub-networks in a single process. It incorporates three modules—pathway-choice routing, pathway regularization, and pathway alignment—to ensure randomized data allocation, diversity among shared models, and alignment with independently trained target models. The approach yields substantial reductions in shadow-model training cost while maintaining competitive attack performance across multiple attacks (e.g., LiRA, RMIA) and datasets, including CIFAR-100/10, CINIC-10, and TinyImageNet. By enabling efficient, scalable attack construction, SHAPOOL also supports model auditing and privacy risk assessment in practical deployment scenarios.

Abstract

Machine learning models are often vulnerable to inference attacks that expose sensitive information from their training data. Shadow model technique is commonly employed in such attacks, such as membership inference. However, the need for a large number of shadow models leads to high computational costs, limiting their practical applicability. Such inefficiency mainly stems from the independent training and use of these shadow models. To address this issue, we present a novel shadow pool training framework SHAPOOL, which constructs multiple shared models and trains them jointly within a single process. In particular, we leverage the Mixture-of-Experts mechanism as the shadow pool to interconnect individual models, enabling them to share some sub-networks and thereby improving efficiency. To ensure the shared models closely resemble independent models and serve as effective substitutes, we introduce three novel modules: path-choice routing, pathway regularization, and pathway alignment. These modules guarantee random data allocation for pathway learning, promote diversity among shared models, and maintain consistency with target models. We evaluate SHAPOOL in the context of various membership inference attacks and show that it significantly reduces the computational cost of shadow model construction while maintaining comparable attack performance.

Figures (7)

• Figure 1: (a): Attack performance and (b-d): Factors of shadow models. We evaluate augmented shadow models on LiRA CNC+22. ori denotes attacks using conventional shadow models. FC-$p$ (or Conv-$p$) denotes attacks with augmented models created via neural masking applied to fully connected layers (or convolutional layers). A larger probability $p$ indicates more perturbation in the model weights. $s$ denotes the number of distinct training subsets used for conventional shadow models.
• Figure 2: (a) Solid lines represent hard connections within shadow models, while dotted lines indicate soft connections across experts in MoE. (b) The sub-networks in orange denote an activated pathway, while those in blue indicate a reference pathway.
• Figure 3: Hyperparameter settings on the LiRA-offline using ResNet18 and CIFAR100.
• Figure 4: Example of a vallina MoE. For simplicity, non-expert layers are omitted here. An activated end-to-end pathway, highlighted in blue, is formed by the experts selected through the input-dependent routing function $\mathcal{R}$, along with the fixed non-expert layers.
• Figure 5: (a) Distribution of shared models of a vanilla MoE. (b) Distribution of shared models after pathway alignment. We present the model output (i.e., scaled logits) distributions for both training (member) and test (non-member) data points using ResNet18 and CIFAR100. The red and blue curves represent the distributions of the target model, while the pink and sky-blue curves illustrate that of shared models from a shadow pool. Since the target model is trained independently, we align the outputs of shared models to better resemble it through pathway alignment.