Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ConvBench: A Comprehensive Benchmark for 2D Convolution Primitive Evaluation

Lucas Alvarenga, Victor Ferrari, Rafael Souza, Marcio Pereira, Guido Araujo

TL;DR

ConvBench tackles the challenge of fairly and comprehensively benchmarking 2D convolution algorithms by constructing a real-world operation set of 9243 convolutions from 1097 models and introducing a Timing Measurement Tool to isolate all preprocessing, execution, and postprocessing steps. The framework demonstrates how Sliced Convolution (SConv) compares to Im2col-GEMM across a broad spectrum of operations, revealing that while SConv outperforms the baseline in most regular convolutions, its packing step is a critical bottleneck in several cases. The study provides actionable diagnostics, such as that packing slowdowns are prevalent when stride differs from 1 and channel counts are small, guiding optimization efforts for convolution designers. By offering detailed timing breakdowns and visualizations, ConvBench enables developers to identify optimization opportunities and to compare diverse convolution algorithms on a fair, consistent basis. Future work aims to extend TM-Tool for richer profiling and to evaluate ConvBench across additional hardware architectures.

Abstract

Convolution is a compute-intensive operation placed at the heart of Convolution Neural Networks (CNNs). It has led to the development of many high-performance algorithms, such as Im2col-GEMM, Winograd, and Direct-Convolution. However, the comparison of different convolution algorithms is an error-prone task as it requires specific data layouts and system resources. Failure to address these requirements might lead to unwanted time penalties. Thus, considering all processing steps within convolution algorithms is essential to comprehensively evaluate and fairly compare their performance. Furthermore, most known convolution benchmarking adopts ad-hoc testing suites with limited coverage and handmade operations. This paper proposes ConvBench, a primitive-level benchmark for the evaluation and comparison of convolution algorithms. It assesses 9243 convolution operations derived from 1097 real-world deep learning models, resulting in performance and execution breakdown graphs for a detailed evaluation. ConvBench capability is evaluated across the Sliced Convolution (SConv) algorithm. The experiments showed results faster than Im2col-GEMM in 93.6% of the convolutions. However, the use of ConvBench allowed the delving into the remaining 6.4% underperforming convolutions, uncovering a critical slowdown of 79.5% on average of SConv's packing step. This analysis underscores a potential source of optimization for SConv, opening up new paths for convolution designers to improve their algorithms.

ConvBench: A Comprehensive Benchmark for 2D Convolution Primitive Evaluation

TL;DR

ConvBench tackles the challenge of fairly and comprehensively benchmarking 2D convolution algorithms by constructing a real-world operation set of 9243 convolutions from 1097 models and introducing a Timing Measurement Tool to isolate all preprocessing, execution, and postprocessing steps. The framework demonstrates how Sliced Convolution (SConv) compares to Im2col-GEMM across a broad spectrum of operations, revealing that while SConv outperforms the baseline in most regular convolutions, its packing step is a critical bottleneck in several cases. The study provides actionable diagnostics, such as that packing slowdowns are prevalent when stride differs from 1 and channel counts are small, guiding optimization efforts for convolution designers. By offering detailed timing breakdowns and visualizations, ConvBench enables developers to identify optimization opportunities and to compare diverse convolution algorithms on a fair, consistent basis. Future work aims to extend TM-Tool for richer profiling and to evaluate ConvBench across additional hardware architectures.

Abstract

Convolution is a compute-intensive operation placed at the heart of Convolution Neural Networks (CNNs). It has led to the development of many high-performance algorithms, such as Im2col-GEMM, Winograd, and Direct-Convolution. However, the comparison of different convolution algorithms is an error-prone task as it requires specific data layouts and system resources. Failure to address these requirements might lead to unwanted time penalties. Thus, considering all processing steps within convolution algorithms is essential to comprehensively evaluate and fairly compare their performance. Furthermore, most known convolution benchmarking adopts ad-hoc testing suites with limited coverage and handmade operations. This paper proposes ConvBench, a primitive-level benchmark for the evaluation and comparison of convolution algorithms. It assesses 9243 convolution operations derived from 1097 real-world deep learning models, resulting in performance and execution breakdown graphs for a detailed evaluation. ConvBench capability is evaluated across the Sliced Convolution (SConv) algorithm. The experiments showed results faster than Im2col-GEMM in 93.6% of the convolutions. However, the use of ConvBench allowed the delving into the remaining 6.4% underperforming convolutions, uncovering a critical slowdown of 79.5% on average of SConv's packing step. This analysis underscores a potential source of optimization for SConv, opening up new paths for convolution designers to improve their algorithms.
Paper Structure (12 sections, 3 figures)

This paper contains 12 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Convolution Benchmarking Tool: ConvBench
  4. The Timing Measurement Tool -- TM-Tool
  5. Convolution Operation Set Construction and Filtering
  6. ConvBench Usage
  7. Result Acquisition and Visualization
  8. Experiments and Results
  9. Experimental procedure
  10. Operation juxtaposing analysis
  11. Results
  12. Conclusion

Figures (3)

  • Figure 1: Proposed Time Measurement Tool. The name of the class start/update API methods follows the same processing step names.
  • Figure 2: In-Convolution speedup/slowdown analysis of Sconv against Im2col-GEMM. (a) refers to the filtered convSet only containing regular operations without zero-padded operations, while (b) refers to the pointwise filtered convSet.
  • Figure 3: Execution breakdown graph. The X-axis indicates the operation index within the convSet, while the Y-axis shows the time magnitude in milliseconds. Particularly, For a given index, Left-sided bars refer to SConv measured timings, and right-sided bars refer to Im2col+GEMM baseline timings.