Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational Positive-incentive Noise: How Noise Benefits Models

Hongyuan Zhang, Sida Huang, Yubin Guo, Xuelong Li

TL;DR

The paper tackles the counterintuitive idea that noise can aid learning by formalizing Positive-incentive Noise (Pi-Noise) and introducing Variational Pi-Noise (VPN). VPN uses a neural noise generator to produce per-sample Gaussian perturbations optimized via a variational lower bound on the mutual information $I(\mathcal{T}, \mathcal{E})$, without altering the base model architecture. The approach employs the reparameterization trick and Monte Carlo estimation to train the generator and optionally the base model, enabling noise to assist both training and inference. Experiments across Fashion-MNIST, CIFAR-10, Tiny ImageNet, and ImageNet demonstrate consistent improvements over baselines and random noise, with visualizations showing the learned noise suppressing irrelevant background regions to reduce the conditional task entropy $H(\mathcal{T}|\mathcal{E})$.

Abstract

A large number of works aim to alleviate the impact of noise due to an underlying conventional assumption of the negative role of noise. However, some existing works show that the assumption does not always hold. In this paper, we investigate how to benefit the classical models by random noise under the framework of Positive-incentive Noise (Pi-Noise). Since the ideal objective of Pi-Noise is intractable, we propose to optimize its variational bound instead, namely variational Pi-Noise (VPN). With the variational inference, a VPN generator implemented by neural networks is designed for enhancing base models and simplifying the inference of base models, without changing the architecture of base models. Benefiting from the independent design of base models and VPN generators, the VPN generator can work with most existing models. From the experiments, it is shown that the proposed VPN generator can improve the base models. It is appealing that the trained variational VPN generator prefers to blur the irrelevant ingredients in complicated images, which meets our expectations.

Variational Positive-incentive Noise: How Noise Benefits Models

TL;DR

The paper tackles the counterintuitive idea that noise can aid learning by formalizing Positive-incentive Noise (Pi-Noise) and introducing Variational Pi-Noise (VPN). VPN uses a neural noise generator to produce per-sample Gaussian perturbations optimized via a variational lower bound on the mutual information , without altering the base model architecture. The approach employs the reparameterization trick and Monte Carlo estimation to train the generator and optionally the base model, enabling noise to assist both training and inference. Experiments across Fashion-MNIST, CIFAR-10, Tiny ImageNet, and ImageNet demonstrate consistent improvements over baselines and random noise, with visualizations showing the learned noise suppressing irrelevant background regions to reduce the conditional task entropy .

Abstract

A large number of works aim to alleviate the impact of noise due to an underlying conventional assumption of the negative role of noise. However, some existing works show that the assumption does not always hold. In this paper, we investigate how to benefit the classical models by random noise under the framework of Positive-incentive Noise (Pi-Noise). Since the ideal objective of Pi-Noise is intractable, we propose to optimize its variational bound instead, namely variational Pi-Noise (VPN). With the variational inference, a VPN generator implemented by neural networks is designed for enhancing base models and simplifying the inference of base models, without changing the architecture of base models. Benefiting from the independent design of base models and VPN generators, the VPN generator can work with most existing models. From the experiments, it is shown that the proposed VPN generator can improve the base models. It is appealing that the trained variational VPN generator prefers to blur the irrelevant ingredients in complicated images, which meets our expectations.
Paper Structure (15 sections, 16 equations, 6 figures, 3 tables, 2 algorithms)

This paper contains 15 sections, 16 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Difference from Conventional Augmentation
  4. Learnable Augmentation
  5. Variational Inference
  6. Variational Positive-incentive Noise
  7. Define Task Entropy over Distribution
  8. Variational Approximation
  9. Positive-incentive Gaussian $\pi$-Noise
  10. Experiments
  11. Implementation Details
  12. Performace Analysis
  13. Visualization
  14. Future Works
  15. Conclusion

Figures (6)

  • Figure 1: Visualization of generated $\pi$-noise. The first line is the original image, the second line shows the generated noise, the third line is the heatmap of variance related to each pixel, and the bottom line is the image with $\pi$-noise. In the fifth image labeled by butterfly, the flower which is most similar to butterfly is disturbed by intense noise so that the recognition task is significantly simplified.
  • Figure 2: The illustration of VPN framework, which consists of a base model and a $\pi$-noise generator. The generator can be trained either with the base model or after the base model. Any model that can predict $p(y | \bm x)$ could be a valid base model.
  • Figure 3: Convergence curve on Fashion-MNIST, CIFAR-10, and Tiny ImageNet, respectively. The base model(ResNet18) and the generator (DNN3) are jointly trained.
  • Figure 4: Impact of noise size $m$ on Fashion-MNIST and CIFAR-10. The base model and generator are ResNet18 and DNN3, respectively.
  • Figure 5: More visualization of generated $\pi$-noise on Tiny ImageNet. The four lines are same as Figure \ref{['figure_visualization']}.
  • ...and 1 more figures