Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detection

Marco Rando, Luca Demetrio, Lorenzo Rosasco, Fabio Roli

TL;DR

This paper tackles adversarial EXEmples for Windows malware detectors under strict functionality constraints by recasting perturbations as a zeroth-order optimization problem in $\mathbb{R}^d$. It introduces a formal manipulation framework with encoding/decoding that enables gradient-free methods, and presents ZEXE, an adaptive zeroth-order attack using structured finite differences, random orthogonal directions, and adaptive exploration. The authors provide theoretical guarantees for non-convex, non-smooth targets and derive a concrete convergence rate for the smoothed objective $F_h$, along with practical parameter guidance. Empirically, ZEXE achieves higher evasion rates than state-of-the-art GAMMA while using substantially smaller perturbations on MalConv and EMBER detectors, demonstrating a meaningful advance for robustness evaluation and security testing in malware detection. The work also explores high-dimensional perturbations and alternative exploration strategies (Langevin, GA), highlighting practical trade-offs between query budgets, perturbation size, and evasion success.

Abstract

Machine learning malware detectors are vulnerable to adversarial EXEmples, i.e., carefully-crafted Windows programs tailored to evade detection. Unlike other adversarial problems, attacks in this context must be functionality-preserving, a constraint that is challenging to address. As a consequence, heuristic algorithms are typically used, which inject new content, either randomly-picked or harvested from legitimate programs. In this paper, we show how learning malware detectors can be cast within a zeroth-order optimization framework, which allows incorporating functionality-preserving manipulations. This permits the deployment of sound and efficient gradient-free optimization algorithms, which come with theoretical guarantees and allow for minimal hyper-parameters tuning. As a by-product, we propose and study ZEXE, a novel zeroth-order attack against Windows malware detection. Compared to state-of-the-art techniques, ZEXE provides improvement in the evasion rate, reducing to less than one third the size of the injected content.

A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detection

TL;DR

This paper tackles adversarial EXEmples for Windows malware detectors under strict functionality constraints by recasting perturbations as a zeroth-order optimization problem in . It introduces a formal manipulation framework with encoding/decoding that enables gradient-free methods, and presents ZEXE, an adaptive zeroth-order attack using structured finite differences, random orthogonal directions, and adaptive exploration. The authors provide theoretical guarantees for non-convex, non-smooth targets and derive a concrete convergence rate for the smoothed objective , along with practical parameter guidance. Empirically, ZEXE achieves higher evasion rates than state-of-the-art GAMMA while using substantially smaller perturbations on MalConv and EMBER detectors, demonstrating a meaningful advance for robustness evaluation and security testing in malware detection. The work also explores high-dimensional perturbations and alternative exploration strategies (Langevin, GA), highlighting practical trade-offs between query budgets, perturbation size, and evasion success.

Abstract

Machine learning malware detectors are vulnerable to adversarial EXEmples, i.e., carefully-crafted Windows programs tailored to evade detection. Unlike other adversarial problems, attacks in this context must be functionality-preserving, a constraint that is challenging to address. As a consequence, heuristic algorithms are typically used, which inject new content, either randomly-picked or harvested from legitimate programs. In this paper, we show how learning malware detectors can be cast within a zeroth-order optimization framework, which allows incorporating functionality-preserving manipulations. This permits the deployment of sound and efficient gradient-free optimization algorithms, which come with theoretical guarantees and allow for minimal hyper-parameters tuning. As a by-product, we propose and study ZEXE, a novel zeroth-order attack against Windows malware detection. Compared to state-of-the-art techniques, ZEXE provides improvement in the evasion rate, reducing to less than one third the size of the injected content.
Paper Structure (17 sections, 6 theorems, 36 equations, 6 tables, 1 algorithm)

This paper contains 17 sections, 6 theorems, 36 equations, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Zeroth-Order Optimization of Adversarial EXEmples
  4. Formulation of Adversarial EXEmples attacks
  5. ZEXE: Adaptive Zeroth-Order attack against Windows Malware Detectors
  6. High-dimensional Perturbations and Exploration Strategies.
  7. Theoretical Results
  8. Experiments
  9. Comparison with GAMMA
  10. High-dimensional Perturbations
  11. Experimental Results
  12. Related Work
  13. Contributions, Limitations, and Future Work
  14. Auxiliary Results
  15. Proofs of Theoretical Results
  16. ...and 2 more sections

Key Result

Theorem 3.1

Under asm:lip_cont, let $\omega > 0$. For every $0 \leq \underline{K} < \bar{K}$ such that for every $k =\underline{K}, \cdots, \bar{K}$, $\| g(v_k) \|^2 > \omega$, let $(v_k)_{k = \underline{K}}^{\bar{K}}$ be the sequence generated by algo:zexe from iteration $\underline{K}$ to $\bar{K}$. Then,

Theorems & Definitions (10)

  • Theorem 3.1: Rates for non-smooth non-convex functions
  • Corollary 1
  • Proposition 1: Smoothing properties
  • proof
  • Lemma 1: Approximation Error
  • proof
  • Lemma 2: Smoothing Lemma
  • proof
  • Lemma 3: Function value decrease
  • proof