Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Algorithms for Parallel Shared-Memory Sparse Matrix-Vector Multiplication on Unstructured Matrices

Kobe Bergmans, Karl Meerbergen, Raf Vandebril

TL;DR

This paper addresses the challenge of efficient parallel SpMV on unstructured sparse matrices by comparing three state-of-the-art approaches and introducing six hybrid algorithms that blend blocking, nonzero reordering, and load-balancing strategies. It provides a modern, open-source implementation and a thorough evaluation across UMA and NUMA shared-memory architectures, highlighting how block-based storage formats and nonzero orderings influence cache reuse and memory bandwidth. Key findings show that BCOH-based hybrids outperform state-of-the-art on NUMA for dense matrices, while CSB/CSBH variants excel on UMA for sparse matrices; conversion costs between storage formats substantially affect the practical benefit of adopting a new format, with break-even points depending on machine characteristics and reuse count. The work offers practical guidance for selecting SpMV strategies based on matrix density and hardware, and suggests directions for automatic tuning and broader on-disk storage strategies to support multiple orderings.

Abstract

The sparse matrix-vector (SpMV) multiplication is an important computational kernel, but it is notoriously difficult to execute efficiently. This paper investigates algorithm performance for unstructured sparse matrices, which are more common than ever because of the trend towards large-scale data collection. The development of an SpMV multiplication algorithm for this type of data is hard due to two factors. First, parallel load balancing issues arise because of the unpredictable nonzero structure. Secondly, SpMV multiplication algorithms are inevitably memory-bound because the sparsity causes a low arithmetic intensity. Three state-of-the-art algorithms for parallel SpMV multiplication on shared-memory systems are discussed. Six new hybrid algorithms are developed which combine optimization techniques of the current algorithms. These techniques include parallelization strategies, storage formats, and nonzero orderings. A modern and high-performance implementation of all discussed algorithms is provided as open-source software. Using this implementation the algorithms are compared. Furthermore, SpMV multiplication algorithms require the matrix to be stored in a specific storage format. Therefore, the conversion time between these storage formats is also analyzed. Both tests are performed for multiple unstructured sparse matrices on different machines: two multi-CPU and two single-CPU architectures. We show that one of the newly developed algorithms outperforms the current state-of-the-art by 19% on one of the multi-CPU architectures. When taking conversion time into consideration, we show that 472 SpMV multiplications are needed to cover the cost of converting to a new storage format for one of the hybrid algorithms on a multi-CPU machine.

Algorithms for Parallel Shared-Memory Sparse Matrix-Vector Multiplication on Unstructured Matrices

TL;DR

This paper addresses the challenge of efficient parallel SpMV on unstructured sparse matrices by comparing three state-of-the-art approaches and introducing six hybrid algorithms that blend blocking, nonzero reordering, and load-balancing strategies. It provides a modern, open-source implementation and a thorough evaluation across UMA and NUMA shared-memory architectures, highlighting how block-based storage formats and nonzero orderings influence cache reuse and memory bandwidth. Key findings show that BCOH-based hybrids outperform state-of-the-art on NUMA for dense matrices, while CSB/CSBH variants excel on UMA for sparse matrices; conversion costs between storage formats substantially affect the practical benefit of adopting a new format, with break-even points depending on machine characteristics and reuse count. The work offers practical guidance for selecting SpMV strategies based on matrix density and hardware, and suggests directions for automatic tuning and broader on-disk storage strategies to support multiple orderings.

Abstract

The sparse matrix-vector (SpMV) multiplication is an important computational kernel, but it is notoriously difficult to execute efficiently. This paper investigates algorithm performance for unstructured sparse matrices, which are more common than ever because of the trend towards large-scale data collection. The development of an SpMV multiplication algorithm for this type of data is hard due to two factors. First, parallel load balancing issues arise because of the unpredictable nonzero structure. Secondly, SpMV multiplication algorithms are inevitably memory-bound because the sparsity causes a low arithmetic intensity. Three state-of-the-art algorithms for parallel SpMV multiplication on shared-memory systems are discussed. Six new hybrid algorithms are developed which combine optimization techniques of the current algorithms. These techniques include parallelization strategies, storage formats, and nonzero orderings. A modern and high-performance implementation of all discussed algorithms is provided as open-source software. Using this implementation the algorithms are compared. Furthermore, SpMV multiplication algorithms require the matrix to be stored in a specific storage format. Therefore, the conversion time between these storage formats is also analyzed. Both tests are performed for multiple unstructured sparse matrices on different machines: two multi-CPU and two single-CPU architectures. We show that one of the newly developed algorithms outperforms the current state-of-the-art by 19% on one of the multi-CPU architectures. When taking conversion time into consideration, we show that 472 SpMV multiplications are needed to cover the cost of converting to a new storage format for one of the hybrid algorithms on a multi-CPU machine.

Paper Structure

This paper contains 18 sections, 3 equations, 4 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Conventional sparse matrix storage formats
  3. State-of-the-art
  4. Compressed Sparse Blocks
  5. Row-Distributed Block CO-H
  6. Merge-Based
  7. Hybrid algorithms
  8. Compressed Sparse Blocks
  9. Row-Distributed Block CO-H
  10. Merge-Based
  11. Experimental design
  12. Implementation
  13. Test setup
  14. Results
  15. SpMV multiplication
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Z-Morton curve projected on a two-by-two, four-by-four and eight-by-eight matrix.
  • Figure 1: Average parallel speedup for different machines. The horizontal axis depicts the number of threads used. The vertical axis depicts the speedup.
  • Figure 2: Hilbert curve projected on a two-by-two, four-by-four and eight-by-eight matrix.
  • Figure 3: Visual representation of parallel merge-sort using three threads.