Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Swap-Based Element Selection for Multiplication-Free Dimension Reduction

Nobutaka Ono

TL;DR

A fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input by using the matrix inversion lemma.

Abstract

In this paper, we propose a fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input. Dimension reduction is a fundamental technique for reducing unnecessary model parameters, mitigating overfitting, and accelerating training and inference. A standard approach is principal component analysis (PCA), but PCA relies on matrix multiplications; on resource-constrained systems, the multiplication count itself can become a bottleneck. Element selection eliminates this cost because the reduction consists only of selecting elements, and thus the key challenge is to determine which elements should be retained. We evaluate a candidate subset through the minimum mean-squared error of linear regression that predicts a target vector from the selected elements, where the target may be, for example, a one-hot label vector in classification. When an explicit target is unavailable, the input itself can be used as the target, yielding a reconstruction-based criterion. The resulting optimization is combinatorial, and exhaustive search is impractical. To address this, we derive an efficient formula for the objective change caused by swapping a selected and an unselected element, using the matrix inversion lemma, and we perform a swap-based local search that repeatedly applies objective-decreasing swaps until no further improvement is possible. Experiments on MNIST handwritten-digit images demonstrate the effectiveness of the proposed method.

Fast Swap-Based Element Selection for Multiplication-Free Dimension Reduction

TL;DR

A fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input by using the matrix inversion lemma.

Abstract

In this paper, we propose a fast algorithm for element selection, a multiplication-free form of dimension reduction that produces a dimension-reduced vector by simply selecting a subset of elements from the input. Dimension reduction is a fundamental technique for reducing unnecessary model parameters, mitigating overfitting, and accelerating training and inference. A standard approach is principal component analysis (PCA), but PCA relies on matrix multiplications; on resource-constrained systems, the multiplication count itself can become a bottleneck. Element selection eliminates this cost because the reduction consists only of selecting elements, and thus the key challenge is to determine which elements should be retained. We evaluate a candidate subset through the minimum mean-squared error of linear regression that predicts a target vector from the selected elements, where the target may be, for example, a one-hot label vector in classification. When an explicit target is unavailable, the input itself can be used as the target, yielding a reconstruction-based criterion. The resulting optimization is combinatorial, and exhaustive search is impractical. To address this, we derive an efficient formula for the objective change caused by swapping a selected and an unselected element, using the matrix inversion lemma, and we perform a swap-based local search that repeatedly applies objective-decreasing swaps until no further improvement is possible. Experiments on MNIST handwritten-digit images demonstrate the effectiveness of the proposed method.
Paper Structure (19 sections, 31 equations, 4 figures, 1 algorithm)

This paper contains 19 sections, 31 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work and motivation.
  3. Contribution.
  4. Problem Formulation
  5. Element Selection as Multiplication-Free Dimension Reduction
  6. Derivation of the Objective Function
  7. Fast Swap-Based Optimization
  8. Exhaustive Search
  9. Objective Reformulation for Efficient Evaluation
  10. Fast Swap-Based Element Selection
  11. Implementation note.
  12. Experiments
  13. Experimental Conditions
  14. Covariance Structure Before and After Selection
  15. Change in Normalized Loss
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Covariance matrices before (left) and after (right) element selection. For display, each entry is transformed as $|\cdot|^{0.3}$.
  • Figure 2: Change in normalized loss (reconstruction loss in our setting). The horizontal line indicates the loss of PCA.
  • Figure 3: Pixels selected by variance (left) and by the proposed method (right).
  • Figure 4: Examples of reconstruction from low-dimensional vectors. From left: original, variance-based selection, random projection, PCA, and the proposed selection.