Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Inference of Networks in the Frequency Domain

Chenqiu Zhao, Guanfang Dong, Anup Basu

TL;DR

This work addresses accelerating neural network inference by operating entirely in the frequency domain for networks trained with frequency regularization. By introducing a frequency inference chain that is dual to spatial-domain computations and by applying non-linear layers directly in the frequency domain, the method enables end-to-end inference with only one $DCT$ at the input and one $IDCT$ at the output. Empirical results on CIFAR-10 across multiple architectures show state-of-the-art performance at high speedups (up to ~150x), and multimedia applications demonstrate practical gains in segmentation accuracy and mobile deployment. The approach highlights a viable path for extreme-speed inference by leveraging sparsity in the frequency domain while maintaining compatibility with standard spatial-domain architectures.

Abstract

It has been demonstrated that networks' parameters can be significantly reduced in the frequency domain with a very small decrease in accuracy. However, given the cost of frequency transforms, the computational complexity is not significantly decreased. In this work, we propose performing network inference in the frequency domain to speed up networks whose frequency parameters are sparse. In particular, we propose a frequency inference chain that is dual to the network inference in the spatial domain. In order to handle the non-linear layers, we make a compromise to apply non-linear operations on frequency data directly, which works effectively. Enabled by the frequency inference chain and the strategy for non-linear layers, the proposed approach completes the entire inference in the frequency domain. Unlike previous approaches which require extra frequency or inverse transforms for all layers, the proposed approach only needs the frequency transform and its inverse once at the beginning and once at the end of a network. Comparisons with state-of-the-art methods demonstrate that the proposed approach significantly improves accuracy in the case of a high speedup ratio (over 100x). The source code is available at \url{https://github.com/guanfangdong/FreqNet-Infer}.

Accelerating Inference of Networks in the Frequency Domain

TL;DR

This work addresses accelerating neural network inference by operating entirely in the frequency domain for networks trained with frequency regularization. By introducing a frequency inference chain that is dual to spatial-domain computations and by applying non-linear layers directly in the frequency domain, the method enables end-to-end inference with only one at the input and one at the output. Empirical results on CIFAR-10 across multiple architectures show state-of-the-art performance at high speedups (up to ~150x), and multimedia applications demonstrate practical gains in segmentation accuracy and mobile deployment. The approach highlights a viable path for extreme-speed inference by leveraging sparsity in the frequency domain while maintaining compatibility with standard spatial-domain architectures.

Abstract

It has been demonstrated that networks' parameters can be significantly reduced in the frequency domain with a very small decrease in accuracy. However, given the cost of frequency transforms, the computational complexity is not significantly decreased. In this work, we propose performing network inference in the frequency domain to speed up networks whose frequency parameters are sparse. In particular, we propose a frequency inference chain that is dual to the network inference in the spatial domain. In order to handle the non-linear layers, we make a compromise to apply non-linear operations on frequency data directly, which works effectively. Enabled by the frequency inference chain and the strategy for non-linear layers, the proposed approach completes the entire inference in the frequency domain. Unlike previous approaches which require extra frequency or inverse transforms for all layers, the proposed approach only needs the frequency transform and its inverse once at the beginning and once at the end of a network. Comparisons with state-of-the-art methods demonstrate that the proposed approach significantly improves accuracy in the case of a high speedup ratio (over 100x). The source code is available at \url{https://github.com/guanfangdong/FreqNet-Infer}.
Paper Structure (14 sections, 10 equations, 2 figures, 3 tables)

This paper contains 14 sections, 10 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Problem Statement
  6. Frequency Inference Chain
  7. Non-linear Layer
  8. Experimental results
  9. Comparisons with State-of-the-art Methods
  10. Applications in Multimedia
  11. Segmentation on U-Net
  12. Applications on Mobile Devices
  13. Spatial-frequency domain duality
  14. Conclusion

Figures (2)

  • Figure 1: The proposed approach accelerates network inference in the frequency domain while keeping training in the spatial domain, to respect the fact that most of the network architectures are designed in the spatial domain.
  • Figure 2: Comparison of different neural network speed up methods. The x-axis represents the inference speed-up rate, and the y-axis represents the classification accuracy in the figures. All experiments are conducted on the CIFAR-10 dataset. The top right figure shows the average accuracy across 8 network architectures, while the smaller figures illustrate the results for each model.