PLM4NDV: Minimizing Data Access for Number of Distinct Values Estimation with Pre-trained Language Models

TL;DR

This paper tackles NDV estimation under limited data access by introducing PLM4NDV, a semantic-based approach that leverages schema information through pretrained language models to derive column and table semantics. It combines frozen PLM embeddings of serialized schema text with a learned, table-aware column interaction mechanism and optional sample statistics to predict $\hat{D}$ via a learned estimator, enabling accurate NDV estimates even with sequential data access or no data access at all. Extensive experiments on a large-scale real-world TabLib dataset show PLM4NDV outperforms traditional sketch- and sampling-based baselines across multiple data-access budgets, while maintaining practical training and inference efficiency. The study also explores LLM prompting for NDV estimation, demonstrating the potential and limitations of zero-shot semantic reasoning, and positions semantic-based NDV estimation as a distinct third paradigm in data statistics.

Abstract

Number of Distinct Values (NDV) estimation of a multiset/column is a basis for many data management tasks, especially within databases. Despite decades of research, most existing methods require either a significant amount of samples through uniform random sampling or access to the entire column to produce estimates, leading to substantial data access costs and potentially ineffective estimations in scenarios with limited data access. In this paper, we propose leveraging semantic information, i.e., schema, to address these challenges. The schema contains rich semantic information that can benefit the NDV estimation. To this end, we propose PLM4NDV, a learned method incorporating Pre-trained Language Models (PLMs) to extract semantic schema information for NDV estimation. Specifically, PLM4NDV leverages the semantics of the target column and the corresponding table to gain a comprehensive understanding of the column's meaning. By using the semantics, PLM4NDV reduces data access costs, provides accurate NDV estimation, and can even operate effectively without any data access. Extensive experiments on a large-scale real-world dataset demonstrate the superiority of PLM4NDV over baseline methods. Our code is available at https://github.com/bytedance/plm4ndv.

Figures (5)

• Figure 1: The model architecture of PLM4NDV is illustrated using a hypothetical table. The statistic component is akin to existing sampling-based methods but operates more efficiently and with a reduced volume of data. In addition, accessing data in the table is optional for PLM4NDV.
• Figure 2: Ablation study on semantic features evaluated under the sequential access condition, where "w/o" indicates removing a kind of semantic features from PLM4NDV.
• Figure 3: Hyperparameter study of PLM4NDV.
• Figure 4: Performance comparison of six PLMs as the foundation PLM in PLM4NDV.
• Figure 5: Estimation error variation with the number of replaced rows in the first 100 rows of the synthesis table.