Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Projection-Free CNN Pruning via Frank-Wolfe with Momentum: Sparser Models with Less Pretraining

Hamza ElMokhtar Shili, Natasha Patnaik, Isabelle Ruble, Kathryn Jarjoura, Daniel Suarez Aguirre

TL;DR

The paper investigates projection-free CNN pruning using Frank-Wolfe variants, including a momentum-enhanced approach, to identify sparse subnetworks with minimal pretraining. By framing pruning as a constrained optimization problem and evaluating on MNIST, the study shows that Frank-Wolfe with momentum yields sparser, more accurate networks than simple pruning and standard FW, with only modest inference-time overhead. A key finding is that substantial performance can be achieved after only 1–2 epochs of dense pretraining, suggesting full pretraining may be unnecessary in resource-constrained settings. The work highlights FW-based pruning as a practical tool for efficient CNN compression and sets direction for broader generalization and structured-pruning extensions.

Abstract

We investigate algorithmic variants of the Frank-Wolfe (FW) optimization method for pruning convolutional neural networks. This is motivated by the "Lottery Ticket Hypothesis", which suggests the existence of smaller sub-networks within larger pre-trained networks that perform comparatively well (if not better). Whilst most literature in this area focuses on Deep Neural Networks more generally, we specifically consider Convolutional Neural Networks for image classification tasks. Building on the hypothesis, we compare simple magnitude-based pruning, a Frank-Wolfe style pruning scheme, and an FW method with momentum on a CNN trained on MNIST. Our experiments track test accuracy, loss, sparsity, and inference time as we vary the dense pre-training budget from 1 to 10 epochs. We find that FW with momentum yields pruned networks that are both sparser and more accurate than the original dense model and the simple pruning baselines, while incurring minimal inference-time overhead in our implementation. Moreover, FW with momentum reaches these accuracies after only a few epochs of pre-training, indicating that full pre-training of the dense model is not required in this setting.

Projection-Free CNN Pruning via Frank-Wolfe with Momentum: Sparser Models with Less Pretraining

TL;DR

The paper investigates projection-free CNN pruning using Frank-Wolfe variants, including a momentum-enhanced approach, to identify sparse subnetworks with minimal pretraining. By framing pruning as a constrained optimization problem and evaluating on MNIST, the study shows that Frank-Wolfe with momentum yields sparser, more accurate networks than simple pruning and standard FW, with only modest inference-time overhead. A key finding is that substantial performance can be achieved after only 1–2 epochs of dense pretraining, suggesting full pretraining may be unnecessary in resource-constrained settings. The work highlights FW-based pruning as a practical tool for efficient CNN compression and sets direction for broader generalization and structured-pruning extensions.

Abstract

We investigate algorithmic variants of the Frank-Wolfe (FW) optimization method for pruning convolutional neural networks. This is motivated by the "Lottery Ticket Hypothesis", which suggests the existence of smaller sub-networks within larger pre-trained networks that perform comparatively well (if not better). Whilst most literature in this area focuses on Deep Neural Networks more generally, we specifically consider Convolutional Neural Networks for image classification tasks. Building on the hypothesis, we compare simple magnitude-based pruning, a Frank-Wolfe style pruning scheme, and an FW method with momentum on a CNN trained on MNIST. Our experiments track test accuracy, loss, sparsity, and inference time as we vary the dense pre-training budget from 1 to 10 epochs. We find that FW with momentum yields pruned networks that are both sparser and more accurate than the original dense model and the simple pruning baselines, while incurring minimal inference-time overhead in our implementation. Moreover, FW with momentum reaches these accuracies after only a few epochs of pre-training, indicating that full pre-training of the dense model is not required in this setting.

Paper Structure

This paper contains 26 sections, 5 equations, 3 figures, 1 table, 4 algorithms.

Table of Contents

  1. Problem Statement and Notation
  2. Notation - From Deep Neural Networks to Convolutional Neural Networks.
  3. Three Stage Approach for Neural Network Sparsification
  4. Background
  5. Lottery Ticket Hypothesis
  6. Suitable Feasible Regions
  7. Convolutional Neural Network Pre-training and Pruning
  8. Convolutional Neural Networks
  9. Resource Constrained Neural Networks and Pruning Paradigm
  10. Frank-Wolfe for Pruning
  11. Frank-Wolfe in Machine Learning Context
  12. Pruning Algorithms
  13. Simple Pruning
  14. Frank-Wolfe Pruning
  15. Frank-Wolfe Pruning + Momentum
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Test accuracy (top) and loss (bottom) of pruned and unpruned models as a function of pre-training budget (epochs).
  • Figure 2: Percentage of non-zero parameters in neural networks post-pruning, relative to the unpruned base model, across different pre-training budgets (epochs).
  • Figure 3: Normalized inference time on the test set (lower is better) for each pruning method, as a function of pre-training budget (epochs).