BMTree: Designing, Learning, and Updating Piecewise Space-Filling Curves for Multi-Dimensional Data Indexing
Jiangneng Li, Yuang Liu, Zheng Wang, Gao Cong, Cheng Long, Walid G. Aref, Han Mao Kiah, Bin Cui
TL;DR
This work addresses indexing multi-dimensional data by designing piecewise space-filling curves (SFCs) that adapt to data and workload distributions. It introduces the Bit Merging Tree (BMTree), a binary-tree framework that jointly partitions the space and assigns bit merging patterns (BMPs) to subspaces, with guarantees of injection and monotonicity. A reinforcement-learning approach based on Monte Carlo Tree Search builds BMTree to optimize query performance, using the ScanRange metric as a fast reward proxy. To cope with changing data and query distributions, the paper proposes a partial retraining mechanism guided by distribution-shift scores and optimization potential, achieving substantial performance gains with significantly reduced retraining cost. Extensive experiments in PostgreSQL and RSMI demonstrate BMTree’s superiority over existing SFCs, and the partial retraining approach offers practical, scalable adaptation to shifts in workload and data distribution.
Abstract
Space-filling curves (SFC, for short) have been widely applied to index multi-dimensional data, which first maps the data to one dimension, and then a one-dimensional indexing method, e.g., the B-tree indexes the mapped data. Existing SFCs adopt a single mapping scheme for the whole data space. However, a single mapping scheme often does not perform well on all the data space. In this paper, we propose a new type of SFC called piecewise SFCs that adopts different mapping schemes for different data subspaces. Specifically, we propose a data structure termed the Bit Merging tree (BMTree) that can generate data subspaces and their SFCs simultaneously, and achieve desirable properties of the SFC for the whole data space. Furthermore, we develop a reinforcement learning-based solution to build the BMTree, aiming to achieve excellent query performance. To update the BMTree efficiently when the distributions of data and/or queries change, we develop a new mechanism that achieves fast detection of distribution shifts in data and queries, and enables partial retraining of the BMTree. The retraining mechanism achieves performance enhancement efficiently since it avoids retraining the BMTree from scratch. Extensive experiments show the effectiveness and efficiency of the BMTree with the proposed learning-based methods.