Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faster Repeated Evasion Attacks in Tree Ensembles

Lorenzo Cascioli, Laurens Devos, Ondřej Kuželka, Jesse Davis

TL;DR

This work tackles the expensive task of generating adversarial examples for tree ensembles by exploiting regularities across sequential evasion tasks. It identifies a small subset of features that are frequently perturbed and introduces two strategies, pruned and mixed, to constrain searches to this subset while preserving correctness guarantees for the mixed approach. The authors provide a statistically grounded method to identify relevant features and demonstrate substantial runtime speedups (up to 35x) across diverse datasets and ensemble types, applicable to both exact (Kantchelian MILP) and approximate (Veritas) attacks. The approach enhances efficiency for robustness assessment, empirical robustness, and model hardening, enabling faster, scalable adversarial analysis in high-dimensional settings.

Abstract

Tree ensembles are one of the most widely used model classes. However, these models are susceptible to adversarial examples, i.e., slightly perturbed examples that elicit a misprediction. There has been significant research on designing approaches to construct such examples for tree ensembles. But this is a computationally challenging problem that often must be solved a large number of times (e.g., for all examples in a training set). This is compounded by the fact that current approaches attempt to find such examples from scratch. In contrast, we exploit the fact that multiple similar problems are being solved. Specifically, our approach exploits the insight that adversarial examples for tree ensembles tend to perturb a consistent but relatively small set of features. We show that we can quickly identify this set of features and use this knowledge to speedup constructing adversarial examples.

Faster Repeated Evasion Attacks in Tree Ensembles

TL;DR

This work tackles the expensive task of generating adversarial examples for tree ensembles by exploiting regularities across sequential evasion tasks. It identifies a small subset of features that are frequently perturbed and introduces two strategies, pruned and mixed, to constrain searches to this subset while preserving correctness guarantees for the mixed approach. The authors provide a statistically grounded method to identify relevant features and demonstrate substantial runtime speedups (up to 35x) across diverse datasets and ensemble types, applicable to both exact (Kantchelian MILP) and approximate (Veritas) attacks. The approach enhances efficiency for robustness assessment, empirical robustness, and model hardening, enabling faster, scalable adversarial analysis in high-dimensional settings.

Abstract

Tree ensembles are one of the most widely used model classes. However, these models are susceptible to adversarial examples, i.e., slightly perturbed examples that elicit a misprediction. There has been significant research on designing approaches to construct such examples for tree ensembles. But this is a computationally challenging problem that often must be solved a large number of times (e.g., for all examples in a training set). This is compounded by the fact that current approaches attempt to find such examples from scratch. In contrast, we exploit the fact that multiple similar problems are being solved. Specifically, our approach exploits the insight that adversarial examples for tree ensembles tend to perturb a consistent but relatively small set of features. We show that we can quickly identify this set of features and use this knowledge to speedup constructing adversarial examples.
Paper Structure (25 sections, 3 theorems, 5 equations, 8 figures, 9 tables, 2 algorithms)

This paper contains 25 sections, 3 theorems, 5 equations, 8 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Tree Ensembles
  4. Evasion Attacks
  5. Method
  6. Modifying the Search Procedure
  7. Pruned Approach
  8. Mixed Approach
  9. Identifying Relevant Features
  10. Experiments
  11. Experimental Setup
  12. Q1: Run time
  13. Q2: Scaling Behavior
  14. Q3: Empirical FNR
  15. Related Work
  16. ...and 10 more sections

Key Result

Proposition 3.1

Given normal example $x$ that is correctly classified by the full ensemble $\bm{T}_{\mathit{full}}$. Let $\bm{T}_{\mathit{prun}} = \textsc{Prune}(\bm{T}_{\mathit{full}}, F_{S}, x)$ and $\tilde{x}= \mathcal{A}(\bm{T}_{\mathit{prun}},x,\delta, t_{\max})$ (i.e., $\bm{T}_{\mathit{prun}}(x) \neq \bm{T}_{

Figures (8)

  • Figure 1: Bar plots showing that most attributes are not modified by the majority of adversarial examples (mnist and webspam only). The leftmost bar shows the number of attributes that are never changed by any of the 10,000 generated adversarial examples. The middle bar shows the number of attributes that are modified at least once but at most by 5% of the adversarial examples. The rightmost bar shows the number of frequently modified features.
  • Figure 2: An example tree using two attributes $\textsc{Height}$ and $\textsc{Age}$ (left). Suppose $F_NS=\{\textsc{Age}\}$. Given an example where $\textsc{Age}=55$, we can prune away the internal node splitting on $\textsc{Age}$. In the resulting tree (right), subtree (b) is pruned because it is unreachable given that $\textsc{Age}=55$ and only subtrees (a) and (c) remain.
  • Figure 3: Average run times for 10 000 calls to full, pruned and mixed) for kantchelian (top) and veritas (bottom). Results are given for both XGBoost and random forest for four selected datasets.
  • Figure 4: Generated adversarial examples for base example $x$ from mnist, with both attacks and both full and pruned setting.
  • Figure 5: Run time of full, mixed and pruned settings for varying the max depth (top) and number of estimators in the ensemble (bottom).
  • ...and 3 more figures

Theorems & Definitions (5)

  • Proposition 3.1
  • proof
  • Theorem 3.2
  • proof
  • Theorem 3.3: greene2017exponential