Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PyTond: Efficient Python Data Science on the Shoulders of Databases

Hesam Shahrokhi, Amirali Kaboli, Mahdi Ghorbani, Amir Shaikhha

TL;DR

This work addresses the scalability gap of Python-based data science pipelines by translating Pandas and NumPy workloads into SQL for in-database execution. It introduces TondIR, a Datalog-like intermediate representation, and a decorator-based Python embedding that together enable broad coverage of Pandas/NumPy APIs, including both dense and sparse tensor layouts, followed by targeted SQL code generation and database-level optimizations. The framework demonstrates substantial performance gains over native Python and competitive baselines across TPC-H and hybrid workloads, with further improvements from a suite of optimizations like dead-code elimination and rule inlining. The results indicate that PyTond can effectively leverage modern DBMS query engines to accelerate data science tasks while reducing data movement, with ongoing work to extend support to iterations, recursions, and higher-order tensors.

Abstract

Python data science libraries such as Pandas and NumPy have recently gained immense popularity. Although these libraries are feature-rich and easy to use, their scalability limitations require more robust computational resources. In this paper, we present PyTond, an efficient approach to push the processing of data science workloads down into the database engines that are already known for their big data handling capabilities. Compared to the previous work, by introducing TondIR, our approach can capture a more comprehensive set of workloads and data layouts. Moreover, by doing IR-level optimizations, we generate better SQL code that improves the query processing by the underlying database engine. Our evaluation results show promising performance improvement compared to Python and other alternatives for diverse data science workloads.

PyTond: Efficient Python Data Science on the Shoulders of Databases

TL;DR

This work addresses the scalability gap of Python-based data science pipelines by translating Pandas and NumPy workloads into SQL for in-database execution. It introduces TondIR, a Datalog-like intermediate representation, and a decorator-based Python embedding that together enable broad coverage of Pandas/NumPy APIs, including both dense and sparse tensor layouts, followed by targeted SQL code generation and database-level optimizations. The framework demonstrates substantial performance gains over native Python and competitive baselines across TPC-H and hybrid workloads, with further improvements from a suite of optimizations like dead-code elimination and rule inlining. The results indicate that PyTond can effectively leverage modern DBMS query engines to accelerate data science tasks while reducing data movement, with ongoing work to extend support to iterations, recursions, and higher-order tensors.

Abstract

Python data science libraries such as Pandas and NumPy have recently gained immense popularity. Although these libraries are feature-rich and easy to use, their scalability limitations require more robust computational resources. In this paper, we present PyTond, an efficient approach to push the processing of data science workloads down into the database engines that are already known for their big data handling capabilities. Compared to the previous work, by introducing TondIR, our approach can capture a more comprehensive set of workloads and data layouts. Moreover, by doing IR-level optimizations, we generate better SQL code that improves the query processing by the underlying database engine. Our evaluation results show promising performance improvement compared to Python and other alternatives for diverse data science workloads.
Paper Structure (18 sections, 10 figures, 7 tables)

This paper contains 18 sections, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Overview
  3. Background
  4. Overview of Approach
  5. TondIR
  6. TondIR Design
  7. Python Embedding
  8. Pandas to TondIR
  9. NumPy to TondIR
  10. TondIR to SQL
  11. Putting it All Together
  12. Efficiency
  13. Evaluation
  14. Experiments Setup
  15. End-to-End Benchmarks
  16. ...and 3 more sections

Figures (10)

  • Figure 1: High-level design of PyTond.
  • Figure 2: Example of an end-to-end translation from Python to SQL.
  • Figure 3: Performance results for TPC-H workloads on a single thread.
  • Figure 4: Performance results for TPC-H workloads on 4 threads.
  • Figure 5: Performance results for data science workloads on a single thread.
  • ...and 5 more figures