Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CODED-SMOOTHING: Coding Theory Helps Generalization

Parsa Moradi, Tayyebeh Jahaninezhad, Mohammad Ali Maddah-Ali

TL;DR

The paper addresses generalization and adversarial vulnerability by introducing the coded-smoothing module, a spline-based, coded-computing-inspired regularizer that encodes a batch of $K$ inputs into $N$ coded samples, processes them through a shared computation, and decodes to enforce local higher-order smoothness via a training objective that blends main and coded paths. It also proposes Randomized Coded Inference (RCI) to bolster robustness at inference by random batch permutations, incurring negligible overhead. Empirical results across supervised and unsupervised tasks, including WGAN-GP, demonstrate improved generalization and state-of-the-art adversarial robustness against gradient-based attacks with minimal computational cost. Overall, the approach provides a practical, scalable regularization framework that unifies supervised and unsupervised settings while delivering robustness benefits through randomized inference.

Abstract

We introduce the coded-smoothing module, which can be seamlessly integrated into standard training pipelines, both supervised and unsupervised, to regularize learning and improve generalization with minimal computational overhead. In addition, it can be incorporated into the inference pipeline to randomize the model and enhance robustness against adversarial perturbations. The design of coded-smoothing is inspired by general coded computing, a paradigm originally developed to mitigate straggler and adversarial failures in distributed computing by processing linear combinations of the data rather than the raw inputs. Building on this principle, we adapt coded computing to machine learning by designing an efficient and effective regularization mechanism that encourages smoother representations and more generalizable solutions. Extensive experiments on both supervised and unsupervised tasks demonstrate that coded-smoothing consistently improves generalization and achieves state-of-the-art robustness against gradient-based adversarial attacks.

CODED-SMOOTHING: Coding Theory Helps Generalization

TL;DR

The paper addresses generalization and adversarial vulnerability by introducing the coded-smoothing module, a spline-based, coded-computing-inspired regularizer that encodes a batch of inputs into coded samples, processes them through a shared computation, and decodes to enforce local higher-order smoothness via a training objective that blends main and coded paths. It also proposes Randomized Coded Inference (RCI) to bolster robustness at inference by random batch permutations, incurring negligible overhead. Empirical results across supervised and unsupervised tasks, including WGAN-GP, demonstrate improved generalization and state-of-the-art adversarial robustness against gradient-based attacks with minimal computational cost. Overall, the approach provides a practical, scalable regularization framework that unifies supervised and unsupervised settings while delivering robustness benefits through randomized inference.

Abstract

We introduce the coded-smoothing module, which can be seamlessly integrated into standard training pipelines, both supervised and unsupervised, to regularize learning and improve generalization with minimal computational overhead. In addition, it can be incorporated into the inference pipeline to randomize the model and enhance robustness against adversarial perturbations. The design of coded-smoothing is inspired by general coded computing, a paradigm originally developed to mitigate straggler and adversarial failures in distributed computing by processing linear combinations of the data rather than the raw inputs. Building on this principle, we adapt coded computing to machine learning by designing an efficient and effective regularization mechanism that encourages smoother representations and more generalizable solutions. Extensive experiments on both supervised and unsupervised tasks demonstrate that coded-smoothing consistently improves generalization and achieves state-of-the-art robustness against gradient-based adversarial attacks.

Paper Structure

This paper contains 29 sections, 2 theorems, 9 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Empirical Risk Minimization (ERM)
  3. Coded-Smoothing Module
  4. Architecture
  5. Spline representation.
  6. Training Regularization using the coded-smoothing Module
  7. Coded-Smoothing is a local higher-order smoother
  8. Robust Inference using a Randomized coded-smoothing module
  9. Experiments
  10. Supervised
  11. Unsupervised
  12. Adversarial Robustness
  13. Covariate Shift Robustness
  14. Related Work
  15. Conclusion
  16. ...and 14 more sections

Key Result

Lemma 1

For a coded-smoothing module with $N$ coded samples, we have: for some constant $C$.

Figures (5)

  • Figure 1: 1.a. In a coded computing module, instead of directly computing $f(x_1),\ldots,f(x_K)$, the system computes $f(\tilde{x}_1),\ldots,f(\tilde{x}_N)$, where $N>K$ and each coded input $\tilde{x}_i$ is a unique weighted linear combination of the originals. The desired outputs are then reconstructed via a decoding procedure, yielding approximations $\hat{f}(x_1)\approx f(x_1),\; \ldots,\; \hat{f}(x_K)\approx f(x_K)$.
  • Figure 2: The proposed Coded-Smoothing as a regularization in training: the coded path includes a coded-smoothing module and runs in parallel to the original forward pass and contributes to the training objective.
  • Figure 3: Coded Smoothing versus Mixup
  • Figure 4: The proposed randomized coded inference: $\pi$ represents a random permutation.
  • Figure 5: (a) TinyImageNet validation loss for different methods during training. (b) Comparison of Inception Score (IS) during training on the CIFAR-10 dataset. In WGAN-GP + Coded-Smoothing, the coded-smoothing module is applied to the generator of the GAN architecture.

Theorems & Definitions (2)

  • Lemma 1
  • Theorem 1: moradi2024coded