Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LMG Index: A Robust and Efficient Learned Index Framework for Multi-Dimensional Performance Balance

Yuzhen Chen, Bin Yao

Abstract

Index structures are fundamental for efficient query processing on large-scale datasets. Learned indexes model the indexing process as a prediction problem to overcome the inherent trade-offs of traditional indexes. However, most existing learned indexes optimize only for limited objectives like query latency or space usage, neglecting other practical evaluation dimensions such as update efficiency and stability. Moreover, many learned indexes rely on assumptions about data distributions or workloads, lacking theoretical guarantees when facing unknown or evolving scenarios, which limits their generality in real-world systems. In this paper, we propose LMG, a robust and efficient learned index framework designed for multi-dimensional performance balance. LMG integrates a decoupled routing structure with theoretical $O(1)$ complexity for fixed key types and an optimal error threshold training algorithm that approaches $O(1)$ overhead in practice. Furthermore, the framework enhances query performance by optimizing gap allocation. Extensive evaluations show that our framework achieves competitive or leading performance across all key evaluation dimensions, including bulk loading (up to 7.55$\times$ faster), point queries (up to 1.68$\times$ faster), range queries (up to 11.41$\times$ faster), and mixed read-write throughput (up to 3.50$\times$ faster). Furthermore, LMG ensures robust long-term stability and high space efficiency (up to 6.26$\times$ smaller footprint). These results demonstrate that LMG significantly mitigates the multi-dimensional performance trade-offs often observed in state-of-the-art approaches, offering a balanced and efficient framework.

LMG Index: A Robust and Efficient Learned Index Framework for Multi-Dimensional Performance Balance

Abstract

Index structures are fundamental for efficient query processing on large-scale datasets. Learned indexes model the indexing process as a prediction problem to overcome the inherent trade-offs of traditional indexes. However, most existing learned indexes optimize only for limited objectives like query latency or space usage, neglecting other practical evaluation dimensions such as update efficiency and stability. Moreover, many learned indexes rely on assumptions about data distributions or workloads, lacking theoretical guarantees when facing unknown or evolving scenarios, which limits their generality in real-world systems. In this paper, we propose LMG, a robust and efficient learned index framework designed for multi-dimensional performance balance. LMG integrates a decoupled routing structure with theoretical complexity for fixed key types and an optimal error threshold training algorithm that approaches overhead in practice. Furthermore, the framework enhances query performance by optimizing gap allocation. Extensive evaluations show that our framework achieves competitive or leading performance across all key evaluation dimensions, including bulk loading (up to 7.55 faster), point queries (up to 1.68 faster), range queries (up to 11.41 faster), and mixed read-write throughput (up to 3.50 faster). Furthermore, LMG ensures robust long-term stability and high space efficiency (up to 6.26 smaller footprint). These results demonstrate that LMG significantly mitigates the multi-dimensional performance trade-offs often observed in state-of-the-art approaches, offering a balanced and efficient framework.
Paper Structure (47 sections, 1 theorem, 16 equations, 13 figures, 4 tables, 2 algorithms)

This paper contains 47 sections, 1 theorem, 16 equations, 13 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. The Idea of Learned Index
  4. Design Strategy Analysis of Learned Index
  5. Model Training Strategies
  6. Segments Indexing Strategies
  7. Error Management Strategies
  8. Strategies Selection for Multi-dimensional Performance Balance and Efficiency
  9. LMIndex Overview
  10. Basic Operations
  11. Point Queries and Range Queries
  12. Update
  13. Bulk Load
  14. LMIndex with Gaps (LMG)
  15. Linear Map
  16. ...and 32 more sections

Key Result

lemma 1

Under the dense key layout ($pos_{i+ 1} - pos_i = 1$), for every data point $p_i$ to contribute a unique vertex to the upper boundary or the lower boundary of the feasible convex hull $P$, the sequence of key intervals $\{\Delta t_i\}$ ($\Delta t_i = t_{i+1} - t_i$) must be strictly monotonically in

Figures (13)

  • Figure 1: Main structure of LMIndex. Linear Map is its segment indexing structure. Each cell within a layer represents a fixed key interval, so the depth of the layer is maximized when the interval is $\le 1$. Precise routing to the covering cell is achieved through layer parameters, such as key range and cell count.
  • Figure 2: Key idea of LM: Mapping segment range from the real number field to the integer field, ensuring each segment corresponds to at least one integer cell.
  • Figure 3: Bulk loading latency and static query latency.
  • Figure 4: Static range query latency under varying scan ranges.
  • Figure 5: Index stability and dynamic performance during continuous insertions (0-50M).
  • ...and 8 more figures

Theorems & Definitions (1)

  • lemma 1