Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LITS: An Optimized Learned Index for Strings (An Extended Version)

Yifan Yang, Shimin Chen

TL;DR

This work addresses the difficulty of applying learned indexes to variable-length string keys by introducing LITS, a string-optimized learned index that combines a Hash-enhanced Prefix Table (HPT) for accurate CDF estimation with per-node local linear models, compact leaf nodes, and subtries for hybrid structure selection via PMSS. The approach eliminates costly last-mile searches and leverages trie strengths, achieving up to $2.43\times$ and $2.27\times$ improvements over HOT and ART for point queries while delivering comparable scan performance across diverse datasets. Key contributions include the HPT-based CDF model, the compact leaf node design, and the PMSS-driven hybridization with subtries, validated on 11 string datasets with substantial performance gains. The results suggest practical significance for real-world databases requiring fast string-key indexing, particularly for workloads with heavy point queries and mixed operations.

Abstract

Index is an important component in database systems. Learned indexes have been shown to outperform traditional tree-based index structures for fixed-sized integer or floating point keys. However, the application of the learned solution to variable-length string keys is under-researched. Our experiments show that existing learned indexes for strings fail to outperform traditional string indexes, such as HOT and ART. String keys are long and variable sized, and often contain skewed prefixes, which make the last-mile search expensive, and adversely impact the capability of learned models to capture the skewed distribution of string keys. In this paper, we propose a novel learned index for string keys, LITS (Learned Index with Hash-enhanced Prefix Table and Sub-tries). We propose an optimized learned model, combining a global Hash-enhanced Prefix Table (HPT) and a per-node local linear model to better distinguish string keys. Moreover, LITS exploits compact leaf nodes and hybrid structures with a PMSS model for efficient point and range operations. Our experimental results using eleven string data sets show that LITS achieves up to 2.43x and 2.27x improvement over HOT and ART for point operations, and attains comparable scan performance.

LITS: An Optimized Learned Index for Strings (An Extended Version)

TL;DR

This work addresses the difficulty of applying learned indexes to variable-length string keys by introducing LITS, a string-optimized learned index that combines a Hash-enhanced Prefix Table (HPT) for accurate CDF estimation with per-node local linear models, compact leaf nodes, and subtries for hybrid structure selection via PMSS. The approach eliminates costly last-mile searches and leverages trie strengths, achieving up to and improvements over HOT and ART for point queries while delivering comparable scan performance across diverse datasets. Key contributions include the HPT-based CDF model, the compact leaf node design, and the PMSS-driven hybridization with subtries, validated on 11 string datasets with substantial performance gains. The results suggest practical significance for real-world databases requiring fast string-key indexing, particularly for workloads with heavy point queries and mixed operations.

Abstract

Index is an important component in database systems. Learned indexes have been shown to outperform traditional tree-based index structures for fixed-sized integer or floating point keys. However, the application of the learned solution to variable-length string keys is under-researched. Our experiments show that existing learned indexes for strings fail to outperform traditional string indexes, such as HOT and ART. String keys are long and variable sized, and often contain skewed prefixes, which make the last-mile search expensive, and adversely impact the capability of learned models to capture the skewed distribution of string keys. In this paper, we propose a novel learned index for string keys, LITS (Learned Index with Hash-enhanced Prefix Table and Sub-tries). We propose an optimized learned model, combining a global Hash-enhanced Prefix Table (HPT) and a per-node local linear model to better distinguish string keys. Moreover, LITS exploits compact leaf nodes and hybrid structures with a PMSS model for efficient point and range operations. Our experimental results using eleven string data sets show that LITS achieves up to 2.43x and 2.27x improvement over HOT and ART for point operations, and attains comparable scan performance.
Paper Structure (25 sections, 1 theorem, 7 equations, 19 figures, 4 tables, 3 algorithms)

This paper contains 25 sections, 1 theorem, 7 equations, 19 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Characteristics of String Keys
  4. Existing Indexes Optimized for Strings
  5. LITS
  6. Overview of LITS
  7. Hash-enhanced Prefix Table (HPT)
  8. Compact Leaf Node
  9. LIT Enhanced with Subtries
  10. Cost Analysis
  11. Evaluation
  12. Experimental Setup
  13. Overall Performance
  14. Benefit of HPT
  15. Benefit of Compact Leaf Node
  16. ...and 10 more sections

Key Result

Theorem 3.1

If prefix ${\mathcal{P}}$ appears $n_{{\mathcal{P}}}$ times in the string data set, and the HPT[$hash({\mathcal{P}})$] row sees $d$ occurrences of other prefixes, then

Figures (19)

  • Figure 1: Prefix skewness of string keys.
  • Figure 2: Comparing index structures optimized for strings.
  • Figure 3: Overview of LITS.
  • Figure 4: An illustration of the CDF computation using the HPT for string "bac". (purple: prefix; red: current character)
  • Figure 5: Replacing multiple nodes with a compact leaf node.
  • ...and 14 more figures

Theorems & Definitions (4)

  • Theorem 3.1
  • Definition 3.1: Common Prefix Length
  • Definition 3.2: Partial Key Length
  • Definition 3.3: Group Partial Key Length