Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bringing Data Transformations Near-Memory for Low-Latency Analytics in HTAP Environments

Arthur Bernhardt, David Volz, Sajjad Tamimi, Andreas Koch, Ilia Petrov

TL;DR

The paper addresses the data movement bottleneck in HTAP analytics by exporting OLTP data to open formats. It proposes transforming data near or inside smart storage as near-data processing (nDT) operations, enabling transactionally consistent on-device snapshots and asynchronous co-execution with foreground workloads. The contributions include an NDP-DBMS architecture (neoDBMS) with in-storage transformations from NSM to Arrow, streaming and on-device materialization modes, delta-nDT for reuse, and interfaces like NDBC for multi-engine/system integration. Experimental results illustrate reduced data movement and robust foreground throughput, with clear cost advantages of smart storage over DRAM ($7000/TB vs $500-1000/TB) and reuse opportunities across pipelines and tools such as SciKit-Learn through memory-mapped exposure of Arrow-formatted data.

Abstract

In this paper we propose an approach for executing data transformations near- or in-storage on intelligent storage systems. The currently prevailing approach of extracting the data and then transforming it to a target format suffers degraded performance during transformation and causes heavy data movement. Our results show robust performance of foreground workloads and lower resource contention. Our vision draws architectural opportunities in multi-engine and multi-system settings, as well as for reuse.

Bringing Data Transformations Near-Memory for Low-Latency Analytics in HTAP Environments

TL;DR

The paper addresses the data movement bottleneck in HTAP analytics by exporting OLTP data to open formats. It proposes transforming data near or inside smart storage as near-data processing (nDT) operations, enabling transactionally consistent on-device snapshots and asynchronous co-execution with foreground workloads. The contributions include an NDP-DBMS architecture (neoDBMS) with in-storage transformations from NSM to Arrow, streaming and on-device materialization modes, delta-nDT for reuse, and interfaces like NDBC for multi-engine/system integration. Experimental results illustrate reduced data movement and robust foreground throughput, with clear cost advantages of smart storage over DRAM (500-1000/TB) and reuse opportunities across pipelines and tools such as SciKit-Learn through memory-mapped exposure of Arrow-formatted data.

Abstract

In this paper we propose an approach for executing data transformations near- or in-storage on intelligent storage systems. The currently prevailing approach of extracting the data and then transforming it to a target format suffers degraded performance during transformation and causes heavy data movement. Our results show robust performance of foreground workloads and lower resource contention. Our vision draws architectural opportunities in multi-engine and multi-system settings, as well as for reuse.
Paper Structure (30 sections, 8 figures)

This paper contains 30 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Motivation.
  3. State-of-the-Art Overview.
  4. Contributions.
  5. Outline.
  6. DBMS design for in-storage transformations
  7. Design Overview.
  8. Overview of the Execution Model
  9. NDP/nDT Invocation.
  10. On-device nDT Scheduling/Preparation.
  11. In-situ Navigation and Data Interpretation.
  12. In-situ address resolution
  13. In-situ Snapshot Construction.
  14. In-storage Data Transformations.
  15. nDT On-Device Result Handling.
  16. ...and 15 more sections

Figures (8)

  • Figure 1: (Top) various alternatives for data transformations: by exporting DBMS data and importing it in the target system; directly accessing the source DBMS; and near-storage. (Bottom) performance comparison of these approaches under HTAP settings with simultaneous OLTP workload.
  • Figure 2: neoDBMS NDP architecture and nDT integration.
  • Figure 3: nDT execution.
  • Figure 4: neoDBMS hardware architecture.
  • Figure 5: Impact of transformations on foreground workload.
  • ...and 3 more figures