Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Partial Partial Aggregates

Claude Brisson

Abstract

We introduce partial partial aggregates (PPA), a query optimization technique for distributed engines that pushes only the local compute phase of an aggregate operation through joins. A query that aggregates after a join involves two logical operations, each requiring a network shuffle. Pushing a full aggregate (COMPUTE$\rightarrow$DISTRIBUTE$\rightarrow$MERGE) below the join introduces a third shuffle. In the specific case where the join key is included in the grouping key and the join is FK-PK, the full pushed aggregate can eliminate the top-level aggregate entirely, making it the preferred choice. In all other key configurations, the top aggregate must remain, and the extra shuffle is wasteful. A PPA pushes only COMPUTE, achieving data reduction before the join without the extra shuffle. The technique relies on the distributive property of aggregates and requires accurate NDV estimation for cost-based decisions.

Partial Partial Aggregates

Abstract

We introduce partial partial aggregates (PPA), a query optimization technique for distributed engines that pushes only the local compute phase of an aggregate operation through joins. A query that aggregates after a join involves two logical operations, each requiring a network shuffle. Pushing a full aggregate (COMPUTEDISTRIBUTEMERGE) below the join introduces a third shuffle. In the specific case where the join key is included in the grouping key and the join is FK-PK, the full pushed aggregate can eliminate the top-level aggregate entirely, making it the preferred choice. In all other key configurations, the top aggregate must remain, and the extra shuffle is wasteful. A PPA pushes only COMPUTE, achieving data reduction before the join without the extra shuffle. The technique relies on the distributive property of aggregates and requires accurate NDV estimation for cost-based decisions.

Paper Structure

This paper contains 31 sections, 3 equations.

Table of Contents

  1. Introduction
  2. Background
  3. Distributed Aggregation
  4. Aggregate Pushdown Through Joins
  5. Column Equivalence via Join Predicates
  6. The Extra Shuffle
  7. Key Relationships and Pushdown Strategy
  8. When PA Eliminates the Top Aggregate
  9. When PA Adds an Extra Shuffle
  10. Partial Partial Aggregates
  11. Definition
  12. Eliminating the Extra Shuffle
  13. Correctness
  14. When PPA Is Beneficial
  15. Cost Model
  16. ...and 16 more sections