DumpyOS: A Data-Adaptive Multi-ary Index for Scalable Data Series Similarity Search

TL;DR

Dumpy addresses the scalability and accuracy challenges of data-series similarity search by introducing a data-adaptive multi-ary index that jointly optimizes proximity and compactness. The core ideas are operationalized through adaptive node splitting (with a dedicated objective function), a leaf-packing workflow, and a parallelized architecture (DumpyOS) that exploits multicore CPUs and NVMe SSDs. Variants such as Dumpy-Fuzzy and DumpyOS-F further improve approximate-search accuracy by boundary-aware duplication and query-driven candidate selection without breaking exact-pruning guarantees. Empirical results show substantial gains in index-building efficiency and query quality across synthetic and real datasets, with practical gains in near-millisecond responses for large-scale workloads.

Abstract

Data series indexes are necessary for managing and analyzing the increasing amounts of data series collections that are nowadays available. These indexes support both exact and approximate similarity search, with approximate search providing high-quality results within milliseconds, which makes it very attractive for certain modern applications. Reducing the pre-processing (i.e., index building) time and improving the accuracy of search results are two major challenges. DSTree and the iSAX index family are state-of-the-art solutions for this problem. However, DSTree suffers from long index building times, while iSAX suffers from low search accuracy. In this paper, we identify two problems of the iSAX index family that adversely affect the overall performance. First, we observe the presence of a proximity-compactness trade-off related to the index structure design (i.e., the node fanout degree), significantly limiting the efficiency and accuracy of the resulting index. Second, a skewed data distribution will negatively affect the performance of iSAX. To overcome these problems, we propose Dumpy, an index that employs a novel multi-ary data structure with an adaptive node splitting algorithm and an efficient building workflow. Furthermore, we devise Dumpy-Fuzzy as a variant of Dumpy which further improves search accuracy by proper duplication of series. To fully leverage the potential of modern hardware including multicore CPUs and Solid State Drives (SSDs), we parallelize Dumpy to DumpyOS with sophisticated indexing and pruning-based querying algorithms. An optimized approximate search algorithm, DumpyOS-F which prominently improves the search accuracy without violating the index, is also proposed.

Figures (28)

• Figure 1: (a) and (b) are the PAA, SAX and iSAX representation ($w=3$, $b=3$). (c) and (d) are the node splitting for iSAX-index family in two- and full-ary structure, respectively.
• Figure 2: Illustration of the adverse effect of skewed splits to the intra-node series proximity. Series b and c are similar to one another, while series a is dissimilar to them. In subfigure (a), series a and b are wrongly grouped in node 1-011-0, whereas in subfigure (b), b and c are correctly grouped in node 10-01-01.
• Figure 3: Node size distribution in the first layer on two 100GB datasets ($w=16$).
• Figure 4: Index structure ($w$=4) and building workflow.
• Figure 5: Illustration of building DumpyOS. The inner barriers are inside a loop within a stage and we only show one iteration. The stage barrier is used for synchronization between stages.

Theorems & Definitions (2)

• definition thmcounterdefinition: Data Series
• definition thmcounterdefinition: $k$NN Query