Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning

Xiang Meng, Wenyu Chen, Riade Benbaki, Rahul Mazumder

TL;DR

This paper proposes FALCON, a novel combinatorial-optimization-based framework for network pruning that jointly takes into account model accuracy, FLOPs, and sparsity constraints, and presents a novel algorithm to approximately solve the ILP.

Abstract

The increasing computational demands of modern neural networks present deployment challenges on resource-constrained devices. Network pruning offers a solution to reduce model size and computational cost while maintaining performance. However, most current pruning methods focus primarily on improving sparsity by reducing the number of nonzero parameters, often neglecting other deployment costs such as inference time, which are closely related to the number of floating-point operations (FLOPs). In this paper, we propose FALCON, a novel combinatorial-optimization-based framework for network pruning that jointly takes into account model accuracy (fidelity), FLOPs, and sparsity constraints. A main building block of our approach is an integer linear program (ILP) that simultaneously handles FLOP and sparsity constraints. We present a novel algorithm to approximately solve the ILP. We propose a novel first-order method for our optimization framework which makes use of our ILP solver. Using problem structure (e.g., the low-rank structure of approx. Hessian), we can address instances with millions of parameters. Our experiments demonstrate that FALCON achieves superior accuracy compared to other pruning approaches within a fixed FLOP budget. For instance, for ResNet50 with 20% of the total FLOPs retained, our approach improves the accuracy by 48% relative to state-of-the-art. Furthermore, in gradual pruning settings with re-training between pruning steps, our framework outperforms existing pruning methods, emphasizing the significance of incorporating both FLOP and sparsity constraints for effective network pruning.

FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruning

TL;DR

This paper proposes FALCON, a novel combinatorial-optimization-based framework for network pruning that jointly takes into account model accuracy, FLOPs, and sparsity constraints, and presents a novel algorithm to approximately solve the ILP.

Abstract

The increasing computational demands of modern neural networks present deployment challenges on resource-constrained devices. Network pruning offers a solution to reduce model size and computational cost while maintaining performance. However, most current pruning methods focus primarily on improving sparsity by reducing the number of nonzero parameters, often neglecting other deployment costs such as inference time, which are closely related to the number of floating-point operations (FLOPs). In this paper, we propose FALCON, a novel combinatorial-optimization-based framework for network pruning that jointly takes into account model accuracy (fidelity), FLOPs, and sparsity constraints. A main building block of our approach is an integer linear program (ILP) that simultaneously handles FLOP and sparsity constraints. We present a novel algorithm to approximately solve the ILP. We propose a novel first-order method for our optimization framework which makes use of our ILP solver. Using problem structure (e.g., the low-rank structure of approx. Hessian), we can address instances with millions of parameters. Our experiments demonstrate that FALCON achieves superior accuracy compared to other pruning approaches within a fixed FLOP budget. For instance, for ResNet50 with 20% of the total FLOPs retained, our approach improves the accuracy by 48% relative to state-of-the-art. Furthermore, in gradual pruning settings with re-training between pruning steps, our framework outperforms existing pruning methods, emphasizing the significance of incorporating both FLOP and sparsity constraints for effective network pruning.
Paper Structure (43 sections, 5 theorems, 44 equations, 4 figures, 5 tables, 6 algorithms)

This paper contains 43 sections, 5 theorems, 44 equations, 4 figures, 5 tables, 6 algorithms.

Table of Contents

  1. INTRODUCTION
  2. PROBLEM FORMULATION
  3. ILP-BASED MAGNITUDE PRUNING UNDER NNZ AND FLOP BUDGETS
  4. Integer Linear Programming (ILP) Formulation
  5. Solving the Relaxed Problem
  6. Theoretical Results
  7. OPTIMIZATION FORMULATION FOR PRUNING
  8. Optimization Formulation with NNZ and FLOP Constraints
  9. A Modified DFO Method for Solving \ref{['eq:miqp-main']}
  10. DFO update
  11. NUMERICAL EXPERIMENTS
  12. One-shot Pruning
  13. Accuracy given the FLOP budget
  14. Quantifying the usefulness of having both FLOP and NNZ budget constraints
  15. Gradual Pruning
  16. ...and 28 more sections

Key Result

Theorem 3.1

Making use of Fact ass:layer, each iteration of Algorithm alg:proj takes $O(L(\log p)^2)$ time complexity. Moreover, for $\epsilon>0$, it takes $O(p\log p+ L(\log p)^2\log(1/\varepsilon))$ time to compute a $\varepsilon$-accurate solution of the dual problem eq:lp-dual.

Figures (4)

  • Figure 1: Inference time improvement (measured by FLOPs) vs. memory usage improvement (measured by NNZ) of CHITA benbaki2023fast. Dashed black line denotes equal improvements.
  • Figure 2: Accuracy of the network pruned by FALCON across three scenarios: pure FLOP constraint, pure sparsity constraint, and joint sparsification. The error bar represents the standard error over five runs.
  • Figure 3: Sparsity of each group of Resnet50 pruned under a fixed FLOP budget (20% of total FLOPs) by FALCON across three scenarios: pure FLOP constraint, pure sparsity constraint, and joint sparsification.
  • Figure 4: Sparsity of each group of models pruned under a fixed FLOP budget (20% of total FLOPs) by FALCON across three scenarios: pure FLOP constraint, pure sparsity constraint, and joint sparsification.

Theorems & Definitions (10)

  • Theorem 3.1
  • Theorem 3.2
  • Lemma 4.1
  • proof
  • Lemma A.1
  • proof
  • proof
  • Proposition A.2
  • proof
  • proof