MojoFrame: Dataframe Library in Mojo Language

TL;DR

MojoFrame introduces the first Mojo-native dataframe library designed to execute core relational operations (filtering, join, group-by) using Mojo's tensor-backed data structures and JIT compilation. It employs a cardinality-aware representation to efficiently integrate non-numeric columns, using tensor storage for numeric data and offloaded structures for high-cardinality strings, complemented by row/column indexers that decouple logical and physical layouts. For relational operations, MojoFrame uses a stateless, trait-based filtering approach, a transposed, immutable-key strategy for group-by, and a factorize-then-hash-join method to leverage Mojo's dictionary while avoiding mutable keys. Empirical results show MojoFrame supports all 22 TPC-H queries and select TPC-DS queries, achieving up to 4.60x speedups over Python-based dataframe libraries on UDF-heavy workloads and demonstrating linear scalability with data size and multi-core execution, while identifying bottlenecks in dictionary performance and in native string tensor support as avenues for future work.

Abstract

Mojo is an emerging programming language built on MLIR (Multi-Level Intermediate Representation) and supports JIT (Just-in-Time) compilation. It enables transparent hardware-specific optimizations (e.g., for CPUs and GPUs), while allowing users to express their logic using Python-like user-friendly syntax. Mojo has demonstrated strong performance on tensor operations; however, its capabilities for relational operations (e.g., filtering, join, and group-by aggregation) common in data science workflows, remain unexplored. To date, no dataframe implementation exists in the Mojo ecosystem. In this paper, we introduce the first Mojo-native dataframe library, called MojoFrame, that supports core relational operations and user-defined functions (UDFs). MojoFrame is built on top of Mojo's tensor to achieve fast operations on numeric columns, while utilizing a cardinality-aware approach to effectively integrate non-numeric columns for flexible data representation. To achieve high efficiency, MojoFrame takes significantly different approaches than existing libraries. We show that MojoFrame supports all operations for TPC-H queries and a selection of TPC-DS queries with promising performance, achieving up to 4.60x speedup versus existing dataframe libraries in other programming languages. Nevertheless, there remain optimization opportunities for MojoFrame (and the Mojo language), particularly in in-memory data representation and dictionary operations.

Figures (14)

• Figure 1: MojoFrame (ours) is the first Mojo-native dataframe library. Mojo is a new language with JIT, MLIR, designed for compatibility with heterogeneous hardware (CPU/GPU).
• Figure 2: Mojo adopts a portable, vendor-independent GPU programming model mojo, enabling execution on heterogeneous hardware (NVIDIA, AMD, Apple) without backend-specific code. This allows parts of relational operations suitable for numeric kernels (e.g., aggregations) to be portable.
• Figure 3: MojoFrame data structure. A tensor stores numeric data. Non-numeric columns are either mapped into the tensor or offloaded into lists based on cardinality. The logical and physical layouts are decoupled with row and column indexers.
• Figure 4: MojoFrame's parallelized filtering with stateless lambda functions vs. Pandas' sequential filtering with apply.
• Figure 5: TPC-H Q16 with MojoFrame and Pandas. MojoFrame follows the per-operation, chained method call programming style of popular dataframe libraries.