Partitioning Strategies for Parallel Computation of Flexible Skylines
Emilio De Lorenzis, Davide Martinenghi
TL;DR
This work tackles the computational challenge of flexible skylines on large datasets by introducing a parallel partitioning framework that distributes data across partitions, performs local computations, and merges results in a final phase. It augments the basic pattern with representative filtering and a NoSeq strategy to eliminate the expensive sequential step, implemented in PySpark and evaluated on datasets with up to 10 million points across multiple dimensions. Results show that balanced partitioning (Angular/Sliced) combined with NoSeq yields the best performance for ND, while PO computations benefit from angular-based approaches; NoSeq consistently offers linear-like scalability and robust gains as data size grows. Overall, the approach significantly accelerates flexible skyline queries and provides a practical pathway for scalable, preference-aware data exploration in big data environments.
Abstract
While classical skyline queries identify interesting data within large datasets, flexible skylines introduce preferences through constraints on attribute weights, and further reduce the data returned. However, computing these queries can be time-consuming for large datasets. We propose and implement a parallel computation scheme consisting of a parallel phase followed by a sequential phase, and apply it to flexible skylines. We assess the additional effect of an initial filtering phase to reduce dataset size before parallel processing, and the elimination of the sequential part (the most time-consuming) altogether. All our experiments are executed in the PySpark framework for a number of different datasets of varying sizes and dimensions.