Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Graph Embedding at Scale: Optimizing CPU-GPU-SSD Integration

Zhonggen Li, Xiangyu Ke, Yifan Zhu, Yunjun Gao, Feifei Li

TL;DR

Legend addresses the scalability gap in graph embedding for billion-scale graphs by orchestrating computation across CPU, GPU, and NVMe SSD storage. It introduces a prefetch-friendly partition loading order and a GPU-SSD direct access driver, paired with a GPU-centric computation pipeline that hides I/O latency and maximizes throughput. Across four large datasets, Legend delivers up to $4.8\\times$ speedup over state-of-the-art RAM/disk-based systems and attains competitive single-GPU performance relative to multi-GPU RAM-based systems, demonstrating strong scalability and high GPU utilization. These results underscore the practicality of a low-cost, heterogeneous architecture for scalable graph embedding.

Abstract

Graph embeddings provide continuous vector representations of nodes in a graph, which are widely applicable in community detection, recommendations, and various scientific fields. However, existing graph embedding systems either face scalability challenges due to the high cost of RAM and multiple GPUs, or rely on disk storage at the expense of I/O efficiency. In this paper, we propose Legend, a lightweight heterogeneous system for graph embedding that systematically redefines data management across CPU, GPU, and NVMe SSD resources. Legend is built on a foundation of efficient data placement and retrieval strategies tailored to the unique strengths of each hardware. Key innovations include a prefetch-friendly embedding loading strategy, enabling GPUs to directly prefetch data from SSDs with minimal I/O overhead, and a high-throughput GPU-SSD direct access driver optimized for graph embedding tasks. Furthermore, we propose a customized parallel execution strategy to maximize GPU utilization, ensuring efficient handling of billion-scale datasets. Extensive experiments demonstrate that Legend achieves up to 4.8x speedup compared to state-of-the-art systems. Moreover, Legend exhibits comparable performance on a single GPU to that of the state-of-the-art system using 4 GPUs on the billion-scale dataset.

Efficient Graph Embedding at Scale: Optimizing CPU-GPU-SSD Integration

TL;DR

Legend addresses the scalability gap in graph embedding for billion-scale graphs by orchestrating computation across CPU, GPU, and NVMe SSD storage. It introduces a prefetch-friendly partition loading order and a GPU-SSD direct access driver, paired with a GPU-centric computation pipeline that hides I/O latency and maximizes throughput. Across four large datasets, Legend delivers up to speedup over state-of-the-art RAM/disk-based systems and attains competitive single-GPU performance relative to multi-GPU RAM-based systems, demonstrating strong scalability and high GPU utilization. These results underscore the practicality of a low-cost, heterogeneous architecture for scalable graph embedding.

Abstract

Graph embeddings provide continuous vector representations of nodes in a graph, which are widely applicable in community detection, recommendations, and various scientific fields. However, existing graph embedding systems either face scalability challenges due to the high cost of RAM and multiple GPUs, or rely on disk storage at the expense of I/O efficiency. In this paper, we propose Legend, a lightweight heterogeneous system for graph embedding that systematically redefines data management across CPU, GPU, and NVMe SSD resources. Legend is built on a foundation of efficient data placement and retrieval strategies tailored to the unique strengths of each hardware. Key innovations include a prefetch-friendly embedding loading strategy, enabling GPUs to directly prefetch data from SSDs with minimal I/O overhead, and a high-throughput GPU-SSD direct access driver optimized for graph embedding tasks. Furthermore, we propose a customized parallel execution strategy to maximize GPU utilization, ensuring efficient handling of billion-scale datasets. Extensive experiments demonstrate that Legend achieves up to 4.8x speedup compared to state-of-the-art systems. Moreover, Legend exhibits comparable performance on a single GPU to that of the state-of-the-art system using 4 GPUs on the billion-scale dataset.
Paper Structure (18 sections, 3 theorems, 1 equation, 9 figures, 10 tables, 2 algorithms)

This paper contains 18 sections, 3 theorems, 1 equation, 9 figures, 10 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Graph Embedding Learning
  4. GPU Architecture
  5. Data Access Mechanism of NVMe SSD
  6. Workflow in Legend
  7. Edge Bucket Ordering
  8. Optimizations on GPU Direct Access to SSD
  9. Optimizations on GPU
  10. Experiments
  11. Experiment Settings
  12. Comparisons with Existing Systems
  13. Evaluations of the Workflow
  14. Prefetch-friendly Order
  15. GPU Direct Access to NVMe SSD
  16. ...and 3 more sections

Key Result

Theorem 1

For the buffer capacity of 3, a partition loading order is classified as a Prefetching Supported Order if it satisfies two properties: (1) The partition that has just been swapped in each buffer state will not be immediately evicted in the subsequent buffer state. (2) Any two partitions may appear c

Figures (9)

  • Figure 1: Comparison of system architectures.
  • Figure 2: Partition-based training scheme. $P_i$ denotes the node partition and $[j]$ denotes the calculating order.
  • Figure 3: Workflow of Legend.
  • Figure 4: Partition loading order in Marius. The numbers inside edge buckets denote their calculated order.
  • Figure 5: Order for prefetching in Legend. The numbers inside edge buckets denote their calculated order. The blue color indicates the edge buckets that can be calculated while prefetching the next partition. $E_i$ is the prefetched partition.
  • ...and 4 more figures

Theorems & Definitions (4)

  • Definition 1: Prefetching Supported Order
  • Theorem 1
  • Theorem 2
  • Theorem 3