Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Architectural Neural Backdoors from First Principles

Harry Langford, Ilia Shumailov, Yiren Zhao, Robert Mullins, Nicolas Papernot

TL;DR

Architectural backdoors embed malicious behavior in the network definition, enabling a chosen trigger to bias outputs even after retraining. The authors generalize the first architectural backdoor to arbitrary triggers using a weight-free detector built from logic primitives, and they taxonomy 12 backdoor variants across trigger detection, signal propagation, and signal integration. They show pre- and post-training injection is possible with limited benign accuracy loss when detectors are faint, and they discuss defenses including architectural sandboxing and provenance checks. User studies reveal humans struggle to detect architectural backdoors, while language models demonstrate stronger detection capabilities, underscoring the need for robust defenses in ML pipelines.

Abstract

While previous research backdoored neural networks by changing their parameters, recent work uncovered a more insidious threat: backdoors embedded within the definition of the network's architecture. This involves injecting common architectural components, such as activation functions and pooling layers, to subtly introduce a backdoor behavior that persists even after (full re-)training. However, the full scope and implications of architectural backdoors have remained largely unexplored. Bober-Irizar et al. [2023] introduced the first architectural backdoor; they showed how to create a backdoor for a checkerboard pattern, but never explained how to target an arbitrary trigger pattern of choice. In this work we construct an arbitrary trigger detector which can be used to backdoor an architecture with no human supervision. This leads us to revisit the concept of architecture backdoors and taxonomise them, describing 12 distinct types. To gauge the difficulty of detecting such backdoors, we conducted a user study, revealing that ML developers can only identify suspicious components in common model definitions as backdoors in 37% of cases, while they surprisingly preferred backdoored models in 33% of cases. To contextualize these results, we find that language models outperform humans at the detection of backdoors. Finally, we discuss defenses against architectural backdoors, emphasizing the need for robust and comprehensive strategies to safeguard the integrity of ML systems.

Architectural Neural Backdoors from First Principles

TL;DR

Architectural backdoors embed malicious behavior in the network definition, enabling a chosen trigger to bias outputs even after retraining. The authors generalize the first architectural backdoor to arbitrary triggers using a weight-free detector built from logic primitives, and they taxonomy 12 backdoor variants across trigger detection, signal propagation, and signal integration. They show pre- and post-training injection is possible with limited benign accuracy loss when detectors are faint, and they discuss defenses including architectural sandboxing and provenance checks. User studies reveal humans struggle to detect architectural backdoors, while language models demonstrate stronger detection capabilities, underscoring the need for robust defenses in ML pipelines.

Abstract

While previous research backdoored neural networks by changing their parameters, recent work uncovered a more insidious threat: backdoors embedded within the definition of the network's architecture. This involves injecting common architectural components, such as activation functions and pooling layers, to subtly introduce a backdoor behavior that persists even after (full re-)training. However, the full scope and implications of architectural backdoors have remained largely unexplored. Bober-Irizar et al. [2023] introduced the first architectural backdoor; they showed how to create a backdoor for a checkerboard pattern, but never explained how to target an arbitrary trigger pattern of choice. In this work we construct an arbitrary trigger detector which can be used to backdoor an architecture with no human supervision. This leads us to revisit the concept of architecture backdoors and taxonomise them, describing 12 distinct types. To gauge the difficulty of detecting such backdoors, we conducted a user study, revealing that ML developers can only identify suspicious components in common model definitions as backdoors in 37% of cases, while they surprisingly preferred backdoored models in 33% of cases. To contextualize these results, we find that language models outperform humans at the detection of backdoors. Finally, we discuss defenses against architectural backdoors, emphasizing the need for robust and comprehensive strategies to safeguard the integrity of ML systems.
Paper Structure (7 sections, 2 figures)

This paper contains 7 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Primer on backdoors
  5. Threat Model
  6. Model Architecture Backdoor
  7. Arbitrary Trigger Detector

Figures (2)

  • Figure 1: A visualisation of the differences between the mode of transmission of the trigger signal from input to output. Left shows interleaved path merging and being separated from the main network. Centre shows shared path directly augmenting a representation of the data. Right shows separate path reading and augmenting a later part of the network.
  • Figure 2: The original architectural backdoor design of bober2023architectural. Here the backdoor is using a separate path and highjacks the model in an untargeted way by adding to the latent space. The ONNX model is visualised with Netron.