Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Compiler for Operations on Relations with Bag Semantics

James Dong, Fredrik Kjolstad

TL;DR

This paper tackles the challenge of fusion for complex relational operations under bag semantics by introducing an Abstract Loop Intermediate Representation (ALIR). It couples a coordinate-tree data model with per-layer storage, then uses Iteration Machines (DFAs) to graph the exact loop structure needed to fuse arbitrary operator pipelines. A code generator translates ALIR and IMs into optimized, portable C++ that co-iterates across inputs, enabling deep fusion (including outer and non-equi joins) and producing competitive performance on LSQB and TPC-H benchmarks. The approach demonstrates strong fusion benefits, maintains portability across data structures, and offers a practical path toward full-featured, high-performance query execution.

Abstract

We describe an abstract loop-based intermediate representation that can express fused implementations of relational algebra expressions on sets and bags (multisets). The loops are abstracted away from physical data structures thus making it easier to generate, reason about, and perform optimization like fusion on. The IR supports the natural relational algebra as well as complex operators that are used in production database systems, including outer joins, non-equi joins, and differences. We then show how to compile this IR to efficient C++ code that co-iterates over the physical data structures present in the relational algebra expression. Our approach lets us express fusion across disparate operators, leading to a 3.87x speedup (0.77--12.23x) on selected LSQB benchmarks and worst-case optimal triangle queries. We also demonstrate that our compiler generates code of high quality: it has similar sequential performance to Hyper on TPC-H with a 1.00x speedup (0.38--4.34x) and competitive parallel performance with a 0.61x speedup (0.23--1.80x). Finally, our approach is portable across data structures.

A Compiler for Operations on Relations with Bag Semantics

TL;DR

This paper tackles the challenge of fusion for complex relational operations under bag semantics by introducing an Abstract Loop Intermediate Representation (ALIR). It couples a coordinate-tree data model with per-layer storage, then uses Iteration Machines (DFAs) to graph the exact loop structure needed to fuse arbitrary operator pipelines. A code generator translates ALIR and IMs into optimized, portable C++ that co-iterates across inputs, enabling deep fusion (including outer and non-equi joins) and producing competitive performance on LSQB and TPC-H benchmarks. The approach demonstrates strong fusion benefits, maintains portability across data structures, and offers a practical path toward full-featured, high-performance query execution.

Abstract

We describe an abstract loop-based intermediate representation that can express fused implementations of relational algebra expressions on sets and bags (multisets). The loops are abstracted away from physical data structures thus making it easier to generate, reason about, and perform optimization like fusion on. The IR supports the natural relational algebra as well as complex operators that are used in production database systems, including outer joins, non-equi joins, and differences. We then show how to compile this IR to efficient C++ code that co-iterates over the physical data structures present in the relational algebra expression. Our approach lets us express fusion across disparate operators, leading to a 3.87x speedup (0.77--12.23x) on selected LSQB benchmarks and worst-case optimal triangle queries. We also demonstrate that our compiler generates code of high quality: it has similar sequential performance to Hyper on TPC-H with a 1.00x speedup (0.38--4.34x) and competitive parallel performance with a 0.61x speedup (0.23--1.80x). Finally, our approach is portable across data structures.

Paper Structure

This paper contains 40 sections, 8 theorems, 8 equations, 22 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Overview
  3. Data Model
  4. Relations as Coordinate Trees
  5. Relation Storage
  6. Lookup.
  7. Iteration.
  8. Related Values and Primary Keys
  9. Abstract Loop IR
  10. Loop specification
  11. Virtual inputs
  12. Precomputation and Temporary Relations
  13. Aggregation
  14. Outer joins
  15. Duplicate handling
  16. ...and 25 more sections

Key Result

lemma 1

Given an iteration machine over input set $S$, for every combination of inputs $p \subseteq S$, there exists a unique floor node with label $\mathop{\mathrm{floor}}\nolimits(p)$ such that $\mathop{\mathrm{floor}}\nolimits(p) \subseteq p$ and $\nexists n'.\, \mathop{\mathrm{floor}}\nolimits(p) \subse

Figures (22)

  • Figure 1: Flowchart of our compiler. Square boxes show intermediate representations.
  • Figure 2: Example coordinate tree. Nodes are labeled with their values; the tuples in each subtree is the the multiset for each node. Dotted lines represent the tuples corresponding to each leaf node.
  • Figure 3: Grammar for ALIR. $R$ represents a relation, and $\alpha$ is an attribute.
  • Figure 4: Inner join in ALIR.
  • Figure 5: ALIR for $\sigma_{x^2 + y^2 = r^2}(A) \mathop{\mathrm{\Join}}\limits B$ where the left-hand side is precomputed into a temporary.
  • ...and 17 more figures

Theorems & Definitions (8)

  • lemma 1: Floor function
  • lemma 2: Coherence of transition function
  • lemma 3: Uniqueness of nodes in product construction
  • lemma 4
  • lemma 4: Floor function
  • lemma 4: Coherence of transition function
  • lemma 4: Uniqueness of nodes in product construction
  • lemma 4