Thallus: An RDMA-based Columnar Data Transport Protocol

TL;DR

This work targets the serialization bottleneck in Arrow-based data transfers over TCP/IP RPCs in OLAP-style workloads. It proposes Thallus, an RDMA-based columnar data transport protocol built on Thallium that uses RPC for control and RDMA bulk transfers for zero-copy data movement, enabling batch-wise transmission of Arrow record batches. A DuckDB-backed server path implements a two-stage workflow (init_scan and iterate/do_rdma/finalize) with a zero-copy Arrow reader via the Arrow C Data Interface, and the client reconstructs Arrow batches from RDMA-fetched buffers for downstream output. Empirical results show up to $5.5\times$ transport acceleration and $2.5\times$ end-to-end improvements over a pure RPC baseline, demonstrating RDMA as a viable path to reduce serialization overhead in modern data centers for Arrow-based workloads.

Abstract

The volume of data generated and stored in contemporary global data centers is experiencing exponential growth. This rapid data growth necessitates efficient processing and analysis to extract valuable business insights. In distributed data processing systems, data undergoes exchanges between the compute servers that contribute significantly to the total data processing duration in adequately large clusters, necessitating efficient data transport protocols. Traditionally, data transport frameworks such as JDBC and ODBC have used TCP/IP-over-Ethernet as their underlying network protocol. Such frameworks require serializing the data into a single contiguous buffer before handing it off to the network card, primarily due to the requirement of contiguous data in TCP/IP. In OLAP use cases, this serialization process is costly for columnar data batches as it involves numerous memory copies that hurt data transport duration and overall data processing performance. We study the serialization overhead in the context of a widely-used columnar data format, Apache Arrow, and propose leveraging RDMA to transport Arrow data over Infiniband in a zero-copy manner. We design and implement Thallus, an RDMA-based columnar data transport protocol for Apache Arrow based on the Thallium framework from the Mochi ecosystem, compare it with a purely Thallium RPC-based implementation, and show substantial performance improvements can be achieved by using RDMA for columnar data transport.

Figures (3)

• Figure 1: Control flow diagram showing the high-level operation of Thallus.
• Figure 2: Comparison of data transport duration using Thallus versus Thallium RPC in column selectivity experiments.
• Figure 3: Comparison of query execution duration using Thallus versus Thallium RPC in column selectivity experiments.