Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Attacks on Data Attribution

Xinhe Wang, Pingbang Hu, Junwei Deng, Jiaqi W. Ma

TL;DR

This work addresses the adversarial vulnerability of data attribution methods used for data valuation and compensation. It introduces two attack strategies, Shadow Attack and Outlier Attack, within a formal threat model that leverages either data-distribution knowledge or black-box model queries to inflate an adversary's compensation share. Empirical results in image classification and text generation show substantial compensation inflation, ranging from $185\%$ to $643\%$, highlighting practical security risks. The findings emphasize the need for robust data attribution mechanisms to prevent manipulation in financial and copyright contexts, and point to future directions for defense against adversarial data contributions.

Abstract

Data attribution aims to quantify the contribution of individual training data points to the outputs of an AI model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no systematic research addressing this issue. In this work, we aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's goal and capabilities and proposing principled adversarial attack methods on data attribution. We present two methods, Shadow Attack and Outlier Attack, which generate manipulated datasets to inflate the compensation adversarially. The Shadow Attack leverages knowledge about the data distribution in the AI applications, and derives adversarial perturbations through "shadow training", a technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods - outlier data points are more likely to be influential - and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-based compensation by at least 200%, while the Outlier Attack achieves compensation inflation ranging from 185% to as much as 643%. Our implementation is ready at https://github.com/TRAIS-Lab/adversarial-attack-data-attribution.

Adversarial Attacks on Data Attribution

TL;DR

This work addresses the adversarial vulnerability of data attribution methods used for data valuation and compensation. It introduces two attack strategies, Shadow Attack and Outlier Attack, within a formal threat model that leverages either data-distribution knowledge or black-box model queries to inflate an adversary's compensation share. Empirical results in image classification and text generation show substantial compensation inflation, ranging from to , highlighting practical security risks. The findings emphasize the need for robust data attribution mechanisms to prevent manipulation in financial and copyright contexts, and point to future directions for defense against adversarial data contributions.

Abstract

Data attribution aims to quantify the contribution of individual training data points to the outputs of an AI model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no systematic research addressing this issue. In this work, we aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's goal and capabilities and proposing principled adversarial attack methods on data attribution. We present two methods, Shadow Attack and Outlier Attack, which generate manipulated datasets to inflate the compensation adversarially. The Shadow Attack leverages knowledge about the data distribution in the AI applications, and derives adversarial perturbations through "shadow training", a technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods - outlier data points are more likely to be influential - and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-based compensation by at least 200%, while the Outlier Attack achieves compensation inflation ranging from 185% to as much as 643%. Our implementation is ready at https://github.com/TRAIS-Lab/adversarial-attack-data-attribution.
Paper Structure (57 sections, 5 theorems, 26 equations, 3 figures, 13 tables)

This paper contains 57 sections, 5 theorems, 26 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Data Attribution for Data Valuation and Compensation.
  4. Membership Inference Attack.
  5. Adversarial Example.
  6. The threat model
  7. The data compensation scenario
  8. Periodic Data Contribution.
  9. AI Training and Data Attribution.
  10. Compensation Mechanism.
  11. The adversary
  12. The Objective of the Adversary.
  13. The Capabilities of the Adversary.
  14. The Action Space of the Adversary.
  15. Shadow Attack
  16. ...and 42 more sections

Key Result

Theorem 5.1

Consider a model trained by ERM on a dataset of size $n$ with a smooth loss $\ell$ with respect to model parameters $\theta$. Assume its corresponding influence score $\tau$, gradient $\nabla_\theta \ell (\theta , z)$, and Hessian $\nabla_\theta^2\ell(\theta , z)$ are all bounded, i.e., $\vert \tau where $z_{\text{test} }$ is a test data point while $\{z_i\}_{i=1}^n$ are the training data points

Figures (3)

  • Figure 1: An illustration of the Shadow Attack method. Shadow training datasets $Z^{(i)}$'s are sampled to estimate the compensation share of a set of data points $Z$ if it were contributed to the AI Developer, which can be leveraged to perturb data points in $Z$ to get a higher compensation share.
  • Figure 2: An illustration of the Outlier Attack method. Here, $\ell(Z)$ denotes the loss used by the model $\mathcal{T}(Z_0)$ when evaluated on the dataset $Z$. The data points in $Z$ are perturbed by maximizing the loss $\ell(Z)$ through black-box attack methods designed to generate adversarial examples.
  • Figure 3: Visualization of MNIST (Top) and CIFAR-10 (Bottom), before and after attacks.

Theorems & Definitions (10)

  • Theorem 5.1: Informal
  • Remark 1
  • Theorem A.1
  • Lemma 1
  • proof
  • Lemma 2: Section 2.4, Part III Stewart1990-ru
  • Lemma 3
  • proof
  • proof : Proof of \ref{['thm:neural-network-formal']}
  • Remark 2