Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirical Perturbation Analysis of Linear System Solvers from a Data Poisoning Perspective

Yixin Liu, Arielle Carr, Lichao Sun

TL;DR

This work investigates how the errors in the input data will affect the fitting error and accuracy of the solution from a linear system-solving algorithm under perturbations common in adversarial attacks.

Abstract

The perturbation analysis of linear solvers applied to systems arising broadly in machine learning settings -- for instance, when using linear regression models -- establishes an important perspective when reframing these analyses through the lens of a data poisoning attack. By analyzing solvers' responses to such attacks, this work aims to contribute to the development of more robust linear solvers and provide insights into poisoning attacks on linear solvers. In particular, we investigate how the errors in the input data will affect the fitting error and accuracy of the solution from a linear system-solving algorithm under perturbations common in adversarial attacks. We propose data perturbation through two distinct knowledge levels, developing a poisoning optimization and studying two methods of perturbation: Label-guided Perturbation (LP) and Unconditioning Perturbation (UP). Existing works mainly focus on deriving the worst-case perturbation bound from a theoretical perspective, and the analysis is often limited to specific kinds of linear system solvers. Under the circumstance that the data is intentionally perturbed -- as is the case with data poisoning -- we seek to understand how different kinds of solvers react to these perturbations, identifying those algorithms most impacted by different types of adversarial attacks.

Empirical Perturbation Analysis of Linear System Solvers from a Data Poisoning Perspective

TL;DR

This work investigates how the errors in the input data will affect the fitting error and accuracy of the solution from a linear system-solving algorithm under perturbations common in adversarial attacks.

Abstract

The perturbation analysis of linear solvers applied to systems arising broadly in machine learning settings -- for instance, when using linear regression models -- establishes an important perspective when reframing these analyses through the lens of a data poisoning attack. By analyzing solvers' responses to such attacks, this work aims to contribute to the development of more robust linear solvers and provide insights into poisoning attacks on linear solvers. In particular, we investigate how the errors in the input data will affect the fitting error and accuracy of the solution from a linear system-solving algorithm under perturbations common in adversarial attacks. We propose data perturbation through two distinct knowledge levels, developing a poisoning optimization and studying two methods of perturbation: Label-guided Perturbation (LP) and Unconditioning Perturbation (UP). Existing works mainly focus on deriving the worst-case perturbation bound from a theoretical perspective, and the analysis is often limited to specific kinds of linear system solvers. Under the circumstance that the data is intentionally perturbed -- as is the case with data poisoning -- we seek to understand how different kinds of solvers react to these perturbations, identifying those algorithms most impacted by different types of adversarial attacks.
Paper Structure (22 sections, 7 theorems, 57 equations, 7 figures, 3 tables)

This paper contains 22 sections, 7 theorems, 57 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Linear Systems and Solvers
  5. Overview of Direct and Iterative Solvers
  6. Direct Solvers
  7. Iterative Solvers
  8. Problem Statement and Threat Model
  9. Methodology
  10. Matrix Perturbation as An Optimization Problem
  11. Theoretical Analysis of UP and LP with Gradient Descent Solver
  12. General Forward $\ell_p$-norm Perturbation Error Bound
  13. Experiments
  14. Experimental Setup
  15. Results and Analysis on Direct Solver
  16. ...and 7 more sections

Key Result

Theorem 1

Let $f(w)$ be an objective function that is L-smooth and convex, e.g., $f(w; X) = \frac{1}{2} \| X w - y \|^2$, where $X$ is a data matrix. Denote the function after applying UP as $f{^\prime}(w; X+ \Delta X)$, which is also L-smooth and convex. Assume that UP is applied, increasing the condition nu

Figures (7)

  • Figure 1: Empirical testing of convergence analysis of LP and UP on Gradient Descent (GD). (a) shows the impact of UP on GD convergence steps across noise radii, (b) displays the spectral radii of perturbed $X^\prime$ across noise radii, and (c) shows the solution gap between the perturbed solution and the original optimal solution across noise radii.
  • Figure 2: Spectral radius and convergence iterations for Jacobi, Gauss-Seidel, and SOR methods under different perturbations.
  • Figure 3: GMRES analysis: eigenvalue distribution, the condition number of eigenvectors of a perturbed matrix, and convergence iterations under perturbations.
  • Figure 4: CG analysis: condition number, eigenvalue alignments, and convergence iterations under perturbations.
  • Figure 5: Gradient descent analysis: the smoothness constant $L$ and convergence iterations under perturbations with various learning rates.
  • ...and 2 more figures

Theorems & Definitions (10)

  • Theorem 1: Convergence Rate of $\alpha$-UP for L-smooth and Convex Functions
  • Theorem 2: Lower Bound on Solution Divergence of $\eta$-LP
  • Theorem 3: $\ell_p$ Forward Error Bound
  • Lemma 1
  • Lemma 2
  • proof
  • Theorem : Convergence Rate of $\alpha$-UP for L-smooth and Convex Functions
  • proof
  • Theorem : Lower Bound on Divergence of Solution due to Label-guided Perturbations (LP)
  • proof