Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory

Philip Groet, Joost Hoozemans, Andreas Grapentin, Felix Eberhardt, Zaid Al-Ars, H. Peter Hofstee

TL;DR

This work addresses memory fragmentation and under-utilization in cluster environments by leveraging ThymesisFlow to create cluster-shared memory and extending Apache Arrow to expose distributed, immutable Arrow tables across nodes, enabling zero-copy and zero-serialization sharing. The authors modify Arrow IPC to serialize only table descriptors while data resides in a globally shared memory region, and implement cache-coherence and fixed-address mapping techniques to maintain pointer validity across nodes. Key contributions include a descriptor-only IPC flow, cache-flush protocols, MAP_FIXED memory mappings, a custom Arrow memory manager for nonstandard regions, remote memory allocations, and the ability to span Tables across multiple nodes. Experimental results show substantial overhead during initialization due largely to cacheline flushing, but large data movements are eliminated through metadata-only transfers, with strided remote accesses illustrating the remaining performance trade-offs and demonstrating the approach's potential for scalable shared-memory analytics on clusters.

Abstract

This paper describes a distributed implementation of Apache Arrow that can leverage cluster-shared load-store addressable memory that is hardware-coherent only within each node. The implementation is built on the ThymesisFlow prototype that leverages the OpenCAPI interface to create a shared address space across a cluster. While Apache Arrow structures are immutable, simplifying their use in a cluster shared memory, this paper creates distributed Apache Arrow tables and makes them accessible in each node.

Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory

TL;DR

This work addresses memory fragmentation and under-utilization in cluster environments by leveraging ThymesisFlow to create cluster-shared memory and extending Apache Arrow to expose distributed, immutable Arrow tables across nodes, enabling zero-copy and zero-serialization sharing. The authors modify Arrow IPC to serialize only table descriptors while data resides in a globally shared memory region, and implement cache-coherence and fixed-address mapping techniques to maintain pointer validity across nodes. Key contributions include a descriptor-only IPC flow, cache-flush protocols, MAP_FIXED memory mappings, a custom Arrow memory manager for nonstandard regions, remote memory allocations, and the ability to span Tables across multiple nodes. Experimental results show substantial overhead during initialization due largely to cacheline flushing, but large data movements are eliminated through metadata-only transfers, with strided remote accesses illustrating the remaining performance trade-offs and demonstrating the approach's potential for scalable shared-memory analytics on clusters.

Abstract

This paper describes a distributed implementation of Apache Arrow that can leverage cluster-shared load-store addressable memory that is hardware-coherent only within each node. The implementation is built on the ThymesisFlow prototype that leverages the OpenCAPI interface to create a shared address space across a cluster. While Apache Arrow structures are immutable, simplifying their use in a cluster shared memory, this paper creates distributed Apache Arrow tables and makes them accessible in each node.
Paper Structure (14 sections, 3 figures, 1 table)

This paper contains 14 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Zero-copy, zero-serialization
  4. Cluster shared memory
  5. ThymesisFlow
  6. Apache Arrow
  7. Methodology
  8. Serializing Arrow columnar structure
  9. Flushing CPU caches before invalidity
  10. Preventing Address Translation by mapping regions to the same address on every node
  11. Allocating in custom memory regions
  12. Remote memory allocations
  13. Spanning Tables across nodes
  14. Results

Figures (3)

  • Figure 1: High-level system components. Arrow as a user library, ThymesisFlow connecting the two OpenCAPI busses together allowing for remote memory accesses.
  • Figure 2: Throughput of data transfers over ethernet compared to sharing metadata zero-copy through the extended Apache Arrow interface.
  • Figure 3: Apache Arrow application throughput for strided read accesses to local and remote memory.