Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Activation-Descent Regularization for Input Optimization of ReLU Networks

Hongzhan Yu, Sicun Gao

TL;DR

This work tackles the challenge of input optimization in ReLU networks, where standard gradients fail to account for changes in activation patterns across regions. It introduces Activation-Descent Regularization, which couples optimization in input space with an activation-pattern space via a differentiable sigmoid-surrogate and a Lagrangian objective, guiding descent along activation-aware directions. The approach yields improved local descent, outperforming traditional gradient-based methods in adversarial attacks, image reconstruction with generative models, and action refinement in deep reinforcement learning, with thorough ablations validating each component. This activation-aware framework offers practical benefits for robust optimization in piecewise-linear networks and highlights directions for theories of convergence and scalability to larger architectures.

Abstract

We present a new approach for input optimization of ReLU networks that explicitly takes into account the effect of changes in activation patterns. We analyze local optimization steps in both the input space and the space of activation patterns to propose methods with superior local descent properties. To accomplish this, we convert the discrete space of activation patterns into differentiable representations and propose regularization terms that improve each descent step. Our experiments demonstrate the effectiveness of the proposed input-optimization methods for improving the state-of-the-art in various areas, such as adversarial learning, generative modeling, and reinforcement learning.

Activation-Descent Regularization for Input Optimization of ReLU Networks

TL;DR

This work tackles the challenge of input optimization in ReLU networks, where standard gradients fail to account for changes in activation patterns across regions. It introduces Activation-Descent Regularization, which couples optimization in input space with an activation-pattern space via a differentiable sigmoid-surrogate and a Lagrangian objective, guiding descent along activation-aware directions. The approach yields improved local descent, outperforming traditional gradient-based methods in adversarial attacks, image reconstruction with generative models, and action refinement in deep reinforcement learning, with thorough ablations validating each component. This activation-aware framework offers practical benefits for robust optimization in piecewise-linear networks and highlights directions for theories of convergence and scalability to larger architectures.

Abstract

We present a new approach for input optimization of ReLU networks that explicitly takes into account the effect of changes in activation patterns. We analyze local optimization steps in both the input space and the space of activation patterns to propose methods with superior local descent properties. To accomplish this, we convert the discrete space of activation patterns into differentiable representations and propose regularization terms that improve each descent step. Our experiments demonstrate the effectiveness of the proposed input-optimization methods for improving the state-of-the-art in various areas, such as adversarial learning, generative modeling, and reinforcement learning.
Paper Structure (18 sections, 2 theorems, 19 equations, 12 figures, 2 tables, 1 algorithm)

This paper contains 18 sections, 2 theorems, 19 equations, 12 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problems with the Input Gradients
  5. Activation-Descent Regularization
  6. Augmenting Inputs with Activation Variables
  7. Optimization Objective
  8. Overall Algorithm
  9. Experiments
  10. Adversarial Optimization
  11. Deep Reinforcement Learning
  12. Adversarial Optimization against Neural Policies
  13. Ablation Study
  14. Conclusion
  15. Approximating Piecewise Constant Function with Sigmoid
  16. ...and 3 more sections

Key Result

Proposition 5.1

If $x$ and $\nu$ satisfy constraints in (feasible), then $f(x)=\hat{f}(x,\nu)$.

Figures (12)

  • Figure 1: Value landscape of 2-D ReLU networks with scalar output space. (a) A small ReLU network with 3 hidden neurons. Each solid black line corresponds to one neuron, representing the set of inputs that achieve zero values. (b) The green and red lines demonstrate the trajectories under the optimization of vanilla GD and the proposed algorithm respectively. (c) The scenario where GD optimization gets stuck around a local maxima. The green dots demonstrate the GD steps, and the black dot is the local maxima. (d) The scenario where the input achieves zero value at one neuron. The green and red arrows are ${\nabla_{x}^{-} f(x)}$ and ${\nabla_{x}^{+} f(x)}$ respectively.
  • Figure 1: (a) Average per-iteration runtime benchmarks for experiments in Section \ref{['section::adversarial_attacking']}. (b)-(c) Model architectures and results for ablation experiments in Section \ref{['section::ablation']}. For instance, Model A is a 3-layer ReLU network with the input dimension $d_{n} = 10$, the latent dimensions $d_{1} = d_{2} = 64$, and the output dimension $d_{m} = 1$.
  • Figure 2: Toy example to illustrate the intuition behind optimizing activation pattern variables $\eta$. We consider to maximize a $1$-dimensional piecewise-linear function where each linear segments (e.g. $f_1$-$f_4$ functions) corresponds to one activation pattern. Left: Applying vanilla GD at $x$ quickly gets stuck at the nearby local optima. Right: Optimizing (\ref{['eq::objective']}) over $\eta$ identifies promising changes to $\eta$ that locally optimizes the overall objective. In the above example, the activation pattern underneath $f_1$ will be targeted, as $f_{1}(x) > f_{3}(x) > f_{4}(x)$, despite the inconsistency between the target activation pattern and the current input. Optimizing (\ref{['eq::constraint_loss']}) helps to correct such inconsistencies.
  • Figure 2: Training parameters
  • Figure 3: (a)-(b) Adversarial optimization experiments in Section \ref{['section::adversarial_attacking']} and \ref{['section::attack_drl']}. Grey and pink shading indicate the untargeted and targeted experiments respectively. (c) Attacks are constructed to worsen the performance of well-trained neural policies in FruitBot and CoinRun.
  • ...and 7 more figures

Theorems & Definitions (6)

  • Definition 3.1: Activation Pattern
  • Proposition 5.1
  • Definition 5.2: Steepest Activation-Descent Direction
  • Definition 5.3: Sigmoid-Surrogate Network
  • Proposition 5.4
  • proof