Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GEqO: ML-Accelerated Semantic Equivalence Detection

Brandon Haynes, Rana Alotaibi, Anna Pavlenko, Jyoti Leeka, Alekh Jindal, Yuanyuan Tian

TL;DR

GEqO is proposed, a portable and lightweight machine-learning-based framework for efficiently identifying semantically equivalent computations at scale that yields significant performance gains and finds up to 2x more equivalences than optimizer and signature-based equivalence detection approaches on TPC-DS-like queries.

Abstract

Large scale analytics engines have become a core dependency for modern data-driven enterprises to derive business insights and drive actions. These engines support a large number of analytic jobs processing huge volumes of data on a daily basis, and workloads are often inundated with overlapping computations across multiple jobs. Reusing common computation is crucial for efficient cluster resource utilization and reducing job execution time. Detecting common computation is the first and key step for reducing this computational redundancy. However, detecting equivalence on large-scale analytics engines requires efficient and scalable solutions that are fully automated. In addition, to maximize computation reuse, equivalence needs to be detected at the semantic level instead of just the syntactic level (i.e., the ability to detect semantic equivalence of seemingly different-looking queries). Unfortunately, existing solutions fall short of satisfying these requirements. In this paper, we take a major step towards filling this gap by proposing GEqO, a portable and lightweight machine-learning-based framework for efficiently identifying semantically equivalent computations at scale. GEqO introduces two machine-learning-based filters that quickly prune out nonequivalent subexpressions and employs a semi-supervised learning feedback loop to iteratively improve its model with an intelligent sampling mechanism. Further, with its novel database-agnostic featurization method, GEqO can transfer the learning from one workload and database to another. Our extensive empirical evaluation shows that, on TPC-DS-like queries, GEqO yields significant performance gains-up to 200x faster than automated verifiers-and finds up to 2x more equivalences than optimizer and signature-based equivalence detection approaches.

GEqO: ML-Accelerated Semantic Equivalence Detection

TL;DR

GEqO is proposed, a portable and lightweight machine-learning-based framework for efficiently identifying semantically equivalent computations at scale that yields significant performance gains and finds up to 2x more equivalences than optimizer and signature-based equivalence detection approaches on TPC-DS-like queries.

Abstract

Large scale analytics engines have become a core dependency for modern data-driven enterprises to derive business insights and drive actions. These engines support a large number of analytic jobs processing huge volumes of data on a daily basis, and workloads are often inundated with overlapping computations across multiple jobs. Reusing common computation is crucial for efficient cluster resource utilization and reducing job execution time. Detecting common computation is the first and key step for reducing this computational redundancy. However, detecting equivalence on large-scale analytics engines requires efficient and scalable solutions that are fully automated. In addition, to maximize computation reuse, equivalence needs to be detected at the semantic level instead of just the syntactic level (i.e., the ability to detect semantic equivalence of seemingly different-looking queries). Unfortunately, existing solutions fall short of satisfying these requirements. In this paper, we take a major step towards filling this gap by proposing GEqO, a portable and lightweight machine-learning-based framework for efficiently identifying semantically equivalent computations at scale. GEqO introduces two machine-learning-based filters that quickly prune out nonequivalent subexpressions and employs a semi-supervised learning feedback loop to iteratively improve its model with an intelligent sampling mechanism. Further, with its novel database-agnostic featurization method, GEqO can transfer the learning from one workload and database to another. Our extensive empirical evaluation shows that, on TPC-DS-like queries, GEqO yields significant performance gains-up to 200x faster than automated verifiers-and finds up to 2x more equivalences than optimizer and signature-based equivalence detection approaches.
Paper Structure (32 sections, 3 equations, 15 figures, 6 tables, 1 algorithm)

This paper contains 32 sections, 3 equations, 15 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Overview
  3. Problem definition
  4. GEqO Overview
  5. Detecting an Equivalence Set for a Large Workload
  6. Equivalence Model Filter (EMF) Overview
  7. Complexity Analysis of GEqO Filters
  8. Feature Engineering
  9. Feature Selection
  10. Logical Plan Featurization
  11. Logical Plan Encoding
  12. Instance-Based Encoding
  13. DB-Agnostic Encoding
  14. Scaling to large workloads
  15. Tensor-based extensions
  16. ...and 17 more sections

Figures (15)

  • Figure 1: Two queries that contain semantically-equivalent subexpressions highlighted by shaded boxes.
  • Figure 2: GEqO Overview
  • Figure 3: Instance-based node vector encoding of an SPJ subexpression. Each operator's metadata is converted to its "vector segment"; unrelated segments are set to zero.
  • Figure 4: Symbolized versions of the subexpression pairs highlighted in Figure \ref{['lst:example1']}.
  • Figure 5: Example of converting instance-based to db-agnostic encoding (table segments only)
  • ...and 10 more figures

Theorems & Definitions (2)

  • Definition 2.1: Vector matching filter (VMF)
  • Definition 6.1: SSFL Confidence Level