Deep Learning Service for Efficient Data Distribution Aware Sorting

TL;DR

NN-sort tackles the scaling bottleneck of large-scale sorting by using a data-distribution aware neural approach to map elements to approximate final positions, followed by iterative refinement and a polish step to guarantee correctness. The method introduces a three-phase pipeline (Input, Sorting, Polish) and a cost model that links performance to per-iteration conflict and misorder rates, supporting a tractable complexity profile. Empirical results on synthetic and real-world data show substantial speedups over traditional sorts and reduced reliance on conventional sorting relative to SageDB Sort, demonstrating practical benefits for big-data systems. The work highlights a scalable, distribution-aware sorting paradigm with clear trade-offs and avenues for handling distribution drift.

Abstract

In this paper, we present a neural network-enabled data distribution aware sorting method, coined as NN-sort. Our approach explores the potential of developing deep learning techniques to speed up large-scale sort operations, enabling data distribution aware sorting as a deep learning service. Compared to traditional pairwise comparison-based sorting algorithms, which sort data elements by performing pairwise operations, NN-sort leverages the neural network model to learn the data distribution and uses it to map large-scale data elements into ordered ones. Our experiments demonstrate the significant advantage of using NN-sort. Measurements on both synthetic and real-world datasets show that NN-sort yields 2.18x to 10x performance improvement over traditional sorting algorithms.

Figures (8)

• Figure 1: $$NN-$$sort architecture
• Figure 2: Algorithm $$NN-$$sort
• Figure 3: Algorithm polish
• Figure 4: Example
• Figure 5: Overall performance evaluation