APEX: Probing Neural Networks via Activation Perturbation

TL;DR

APEX introduces Activation Perturbation for EXploration to probe neural representations at inference time by injecting noise after hidden activations while leaving inputs and parameters fixed. The method yields a two-regime picture: in the small-noise regime, prediction stability aligns with sample-level regularity and semantic structure; in the large-noise regime, the stationary output becomes model-dependent and exposes global biases, including backdoor target alignment. The work demonstrates that activation perturbations recover representation-level structure more faithfully than input or parameter perturbations, uncovering both locally stable samples and training-induced global biases, such as backdoor effects, and offering a scalable metric (escape noise) for sample regularity. These insights provide a unified lens for understanding generalization, memorization, robustness, and security vulnerabilities in neural networks, with practical implications for model auditing and analysis. The framework is validated across vision architectures and datasets, showing robustness to noise type and activation function and revealing capacity-driven differences in representation-space allocation.

Abstract

Prior work on probing neural networks primarily relies on input-space analysis or parameter perturbation, both of which face fundamental limitations in accessing structural information encoded in intermediate representations. We introduce Activation Perturbation for EXploration (APEX), an inference-time probing paradigm that perturbs hidden activations while keeping both inputs and model parameters fixed. We theoretically show that activation perturbation induces a principled transition from sample-dependent to model-dependent behavior by suppressing input-specific signals and amplifying representation-level structure, and further establish that input perturbation corresponds to a constrained special case of this framework. Through representative case studies, we demonstrate the practical advantages of APEX. In the small-noise regime, APEX provides a lightweight and efficient measure of sample regularity that aligns with established metrics, while also distinguishing structured from randomly labeled models and revealing semantically coherent prediction transitions. In the large-noise regime, APEX exposes training-induced model-level biases, including a pronounced concentration of predictions on the target class in backdoored models. Overall, our results show that APEX offers an effective perspective for exploring, and understanding neural networks beyond what is accessible from input space alone.

Figures (21)

• Figure 1: Conceptual illustration of probing mechanisms. Input perturbations are constrained by input-to-representation mapping, parameter perturbations modify the model itself, whereas APEX operates directly on hidden representations, enabling exploration beyond input reachability without modifying model parameters.
• Figure 2: Real examples of the output distributions obtained from two "horse" samples.
• Figure 3: Semantic alignment under small noise. Predicted probability of the reassigned class in a controlled CIFAR-10 setup where two classes share the same input distribution. Only activation perturbation induces a monotonic transfer between the two classes, while input and parameter perturbations do not. (The noise magnitude for weight perturbation is rescaled, and the corresponding $\sigma$ values equal the x-axis values multiplied by $10^{-1}$)
• Figure 4: Average escape noise for models with different percentages of samples randomly labeled.
• Figure 5: Relationship between average escape noise and memorization score (Mem-score), consistency score (C-score) on ImageNet and CIFAR-100.

Theorems & Definitions (4)

• Theorem 3.1: Decomposition of activation perturbation
• proof : Decomposition after ReLU with uniformly bounded residual
• Lemma E.1
• proof