Efficient Dataframe Systems: Lazy Fat Pandas on a Diet

TL;DR

LaFP addresses the memory and performance challenges of Pandas on large datasets by providing a drop-in optimization that lets users write in plain Pandas while LaFP automatically rewrites code and executes lazily across multiple backends. It combines Just-in-Time static analysis with a lazy runtime to build and optimize a task graph, enabling back-ends such as Pandas, Dask, and Modin. Key contributions include SCIRPy as a Soot-compatible IR, a JIT static-analysis framework, lazy evaluation with lazy print, and runtime optimizations like predicate pushdown and common computation reuse, all coordinated with metadata-driven decisions. The approach yields substantial speedups (up to 19×) and memory reductions on large datasets, while preserving correctness, illustrating practical impact for scalable data science workloads.

Abstract

Pandas is widely used for data science applications, but users often run into problems when datasets are larger than memory. There are several frameworks based on lazy evaluation that handle large datasets, but the programs have to be rewritten to suit the framework, and the presence of multiple frameworks complicates the life of a programmer. In this paper we present a framework that allows programmers to code in plain Pandas; with just two lines of code changed by the user, our system optimizes the program using a combination of just-in-time static analysis, and runtime optimization based on a lazy dataframe wrapper framework. Moreover, our system allows the programmer to choose the backend. It works seamlessly with Pandas, Dask, and Modin, allowing the choice of the best-suited backend for an application based on factors such as data size. Performance results on a variety of programs show the benefits of our framework.

Figures (15)

• Figure 1: Overview of Our Lazy Fat Pandas System
• Figure 2: Code changes for Using Lazy Fat Pandas Framework
• Figure 3: Sample Program
• Figure 4: Optimized Version of Program in Figure \ref{['l:p1']}
• Figure 5: Just-in-Time Static Analysis