Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Data Chunking in Deduplication Systems using Vector Instructions

Sreeharsha Udayashankar, Abdelrahman Baba, Samer Al-Kiswany

TL;DR

This paper tackles the data chunking bottleneck in deduplication by focusing on hashless CDC algorithms, which are faster but traditionally offer slightly lower space savings. It introduces VectorCDC, a vector-instruction–based framework that accelerates two core patterns—Extreme Byte Search and Range Scan—using a tree-based approach and packed scanning, respectively. The authors implement several variants (AE-Max, AE-Min, MAXP, RAM) and demonstrate substantial throughput improvements (up to 8.35×–26.2× over prior vector-accelerated methods and 15.3×–207.2× over unaccelerated baselines) across Intel, AMD, ARM, and IBM Power CPUs, without sacrificing space savings. The work provides a portable, publicly available solution integrated with DedupBench and offers extensive cross-architecture evaluation, validating that hashless CDC with VectorCDC yields the best overall performance for modern deduplication pipelines.

Abstract

Content-defined Chunking (CDC) algorithms dictate the overall space savings that deduplication systems achieve. However, due to their need to scan each file in its entirety, they are slow and often the main performance bottleneck within data deduplication. We present VectorCDC, a method to accelerate hashless CDC algorithms using vector CPU instructions, such as SSE / AVX. We analyzed the state-of-the-art chunking algorithms and discovered that hashless algorithms primarily use two data processing patterns to identify chunk boundaries: Extreme Byte Searches and Range Scans. VectorCDC presents a vector-friendly approach to accelerate these two patterns. Using VectorCDC, we accelerated three state-of-the-art hashless chunking algorithms: RAM, AE, and MAXP. Our evaluation shows that VectorCDC is effective on Intel, AMD, ARM, and IBM CPUs, achieving 8.35x - 26.2x higher throughput than existing vector-accelerated algorithms, and 15.3x - 207.2x higher throughput than existing unaccelerated algorithms. VectorCDC achieves this without affecting the deduplication space savings.

Accelerating Data Chunking in Deduplication Systems using Vector Instructions

TL;DR

This paper tackles the data chunking bottleneck in deduplication by focusing on hashless CDC algorithms, which are faster but traditionally offer slightly lower space savings. It introduces VectorCDC, a vector-instruction–based framework that accelerates two core patterns—Extreme Byte Search and Range Scan—using a tree-based approach and packed scanning, respectively. The authors implement several variants (AE-Max, AE-Min, MAXP, RAM) and demonstrate substantial throughput improvements (up to 8.35×–26.2× over prior vector-accelerated methods and 15.3×–207.2× over unaccelerated baselines) across Intel, AMD, ARM, and IBM Power CPUs, without sacrificing space savings. The work provides a portable, publicly available solution integrated with DedupBench and offers extensive cross-architecture evaluation, validating that hashless CDC with VectorCDC yields the best overall performance for modern deduplication pipelines.

Abstract

Content-defined Chunking (CDC) algorithms dictate the overall space savings that deduplication systems achieve. However, due to their need to scan each file in its entirety, they are slow and often the main performance bottleneck within data deduplication. We present VectorCDC, a method to accelerate hashless CDC algorithms using vector CPU instructions, such as SSE / AVX. We analyzed the state-of-the-art chunking algorithms and discovered that hashless algorithms primarily use two data processing patterns to identify chunk boundaries: Extreme Byte Searches and Range Scans. VectorCDC presents a vector-friendly approach to accelerate these two patterns. Using VectorCDC, we accelerated three state-of-the-art hashless chunking algorithms: RAM, AE, and MAXP. Our evaluation shows that VectorCDC is effective on Intel, AMD, ARM, and IBM CPUs, achieving 8.35x - 26.2x higher throughput than existing vector-accelerated algorithms, and 15.3x - 207.2x higher throughput than existing unaccelerated algorithms. VectorCDC achieves this without affecting the deduplication space savings.

Paper Structure

This paper contains 44 sections, 2 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Data Chunking
  4. Hash-based algorithms.
  5. Hashless algorithms.
  6. Deduplication Metrics
  7. Space savings.
  8. Chunking throughput.
  9. Chunk size distribution.
  10. Vector Instructions
  11. Intel and AMD
  12. ARM
  13. IBM Power
  14. Motivation
  15. Performance bottlenecks in data deduplication
  16. ...and 29 more sections

Figures (14)

  • Figure 1: Hash-based chunking algorithms
  • Figure 2: Hashless chunking algorithms
  • Figure 3: Time taken for Data Chunking vs Fingerprinting while deduplicating randomized data with FastCDC fastcdc and an 8 KB average chunk size.
  • Figure 4: SS-CDC sscdc: Accelerating the rolling hash phase
  • Figure 5: SS-CDC sscdc throughputs on randomized data with AVX-512 instructions
  • ...and 9 more figures