Computing the Non-Dominated Flexible Skyline in Vertically Distributed Datasets with No Random Access
Davide Martinenghi
TL;DR
This work addresses computing the $ND_k(r;\mathcal{F})$ operator in vertically distributed data with no random access (NRA). It proposes a two-phase NRA algorithm—growing and shrinking phases—driven by a threshold point $\tau$ and $\mathcal{F}$-dominance checks, and provides a formal instance-optimality proof. The paper formalizes relationships among $Top_k$, $Sky$, $ND$, and $ND_k$, and validates the approach with extensive experiments on synthetic (UNI/ANT) and real (NBA) datasets across varied parameters. The results demonstrate strong performance on uniformly distributed data and reveal trade-offs and limitations under challenging distributions, offering practical guidance on algorithmic parameters such as batch size.
Abstract
In today's data-driven world, algorithms operating with vertically distributed datasets are crucial due to the increasing prevalence of large-scale, decentralized data storage. These algorithms enhance data privacy by processing data locally, reducing the need for data transfer and minimizing exposure to breaches. They also improve scalability, as they can handle vast amounts of data spread across multiple locations without requiring centralized access. Top-k queries have been studied extensively under this lens, and are particularly suitable in applications involving healthcare, finance, and IoT, where data is often sensitive and distributed across various sources. Classical top-k algorithms are based on the availability of two kinds of access to sources: sorted access, i.e., a sequential scan in the internal sort order, one tuple at a time, of the dataset; random access, which provides all the information available at a data source for a tuple whose id is known. However, in scenarios where data retrieval costs are high or data is streamed in real-time or, simply, data are from external sources that only offer sorted access, random access may become impractical or impossible, due to latency issues or data access constraints. Fortunately, a long tradition of algorithms designed for the "no random access" (NRA) scenario exists for classical top-k queries. Yet, these do not cover the recent advances in ranking queries, proposing hybridizations of top-k queries (which are preference-aware and control the output size) and skyline queries (which are preference-agnostic and have uncontrolled output size). The non-dominated flexible skyline (ND) is one such proposal. We introduce an algorithm for computing ND in the NRA scenario, prove its correctness and optimality within its class, and provide an experimental evaluation covering a wide range of cases, with both synthetic and real datasets.