Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A new membership inference attack that spots memorization in generative and predictive models: Loss-Based with Reference Model algorithm (LBRM)

Faiz Taleb, Ivan Gazeau, Maryline Laurent

TL;DR

This work tackles privacy risks from unintended memorization in time series imputation by introducing Loss-Based with Reference Model (LBRM), an MIA that uses a reference model to distinguish memorized data from genuine predictions. The method computes a DTW-based loss ratio \\mathcal{R}(x) = \\frac{L_T(x)}{L_R(x)} between a target \\mathcal{T} and a reference \\mathcal{R}, flagging training-data points when \\mathcal{R}(x) \le \\theta. Evaluations on SAITS (predictive) and Autoencoder (generative) architectures across Scenario 1 (unfined-tuned) and Scenario 2 (fine-tuned) show large AUROC gains (roughly 40% without fine-tuning and around 60% with fine-tuning) over Naive Loss baselines, demonstrating robust detection of memorization in time series imputation. The results underscore the method's versatility and potential to mitigate privacy risks in practical deployment, while suggesting future work to refine the reference model and extend LBRM to other generative settings.

Abstract

Generative models can unintentionally memorize training data, posing significant privacy risks. This paper addresses the memorization phenomenon in time series imputation models, introducing the Loss-Based with Reference Model (LBRM) algorithm. The LBRM method leverages a reference model to enhance the accuracy of membership inference attacks, distinguishing between training and test data. Our contributions are twofold: first, we propose an innovative method to effectively extract and identify memorized training data, significantly improving detection accuracy. On average, without fine-tuning, the AUROC improved by approximately 40\%. With fine-tuning, the AUROC increased by approximately 60\%. Second, we validate our approach through membership inference attacks on two types of architectures designed for time series imputation, demonstrating the robustness and versatility of the LBRM approach in different contexts. These results highlight the significant enhancement in detection accuracy provided by the LBRM approach, addressing privacy risks in time series imputation models.

A new membership inference attack that spots memorization in generative and predictive models: Loss-Based with Reference Model algorithm (LBRM)

TL;DR

This work tackles privacy risks from unintended memorization in time series imputation by introducing Loss-Based with Reference Model (LBRM), an MIA that uses a reference model to distinguish memorized data from genuine predictions. The method computes a DTW-based loss ratio \\mathcal{R}(x) = \\frac{L_T(x)}{L_R(x)} between a target \\mathcal{T} and a reference \\mathcal{R}, flagging training-data points when \\mathcal{R}(x) \le \\theta. Evaluations on SAITS (predictive) and Autoencoder (generative) architectures across Scenario 1 (unfined-tuned) and Scenario 2 (fine-tuned) show large AUROC gains (roughly 40% without fine-tuning and around 60% with fine-tuning) over Naive Loss baselines, demonstrating robust detection of memorization in time series imputation. The results underscore the method's versatility and potential to mitigate privacy risks in practical deployment, while suggesting future work to refine the reference model and extend LBRM to other generative settings.

Abstract

Generative models can unintentionally memorize training data, posing significant privacy risks. This paper addresses the memorization phenomenon in time series imputation models, introducing the Loss-Based with Reference Model (LBRM) algorithm. The LBRM method leverages a reference model to enhance the accuracy of membership inference attacks, distinguishing between training and test data. Our contributions are twofold: first, we propose an innovative method to effectively extract and identify memorized training data, significantly improving detection accuracy. On average, without fine-tuning, the AUROC improved by approximately 40\%. With fine-tuning, the AUROC increased by approximately 60\%. Second, we validate our approach through membership inference attacks on two types of architectures designed for time series imputation, demonstrating the robustness and versatility of the LBRM approach in different contexts. These results highlight the significant enhancement in detection accuracy provided by the LBRM approach, addressing privacy risks in time series imputation models.
Paper Structure (25 sections, 6 equations, 2 figures, 3 tables, 1 algorithm)

This paper contains 25 sections, 6 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Definition of Multivariate Time Series Imputation
  5. Membership attack
  6. Exploiting Memorization for Membership Inference Attacks with our LBRM Approach
  7. Constructing the Reference Model for Benchmarking
  8. Evaluating Predictions and Determining Membership
  9. Determining the Threshold Value $\theta$
  10. Performance Evaluation
  11. Experimental Methodology
  12. Metrics.
  13. Datasets.
  14. Naive loss attack.
  15. Experiment Scenarios.
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Result of the membership attack for scenario 1
  • Figure 2: Result of the membership attack for scenario 2 (fine tuning)