LEDD: Large Language Model-Empowered Data Discovery in Data Lakes
Qi An, Chihua Ying, Yuqing Zhu, Yihao Xu, Manwei Zhang, Jianmin Wang
TL;DR
LEDD addresses the challenge of data discovery in expanding data lakes by combining LLM-enabled semantic processing with a federated data lake (IGinX) to generate hierarchical semantic catalogs and support semantic table search. It introduces an end-to-end architecture that uses six facets of metadata, two embedding strategies, and iterative clustering to build a navigable hierarchy, along with real-time relation analysis during exploration. The paper details an extensible Python/UDF-based interface for algorithm replacement and demonstrates the approach on a large schema benchmark, highlighting semantic search, relational insights, and openness for extension. The work offers practical impact for data discovery, model training data selection, schema linking for text-to-SQL tasks, and governance, with open-source potential via GitHub.
Abstract
Data discovery in data lakes with ever increasing datasets has long been recognized as a big challenge in the realm of data management, especially for semantic search of and hierarchical global catalog generation of tables. While large language models (LLMs) facilitate the processing of data semantics, challenges remain in architecting an end-to-end system that comprehensively exploits LLMs for the two semantics-related tasks. In this demo, we propose LEDD, an end-to-end system with an extensible architecture that leverages LLMs to provide hierarchical global catalogs with semantic meanings and semantic table search for data lakes. Specifically, LEDD can return semantically related tables based on natural-language specification. These features make LEDD an ideal foundation for downstream tasks such as model training and schema linking for text-to-SQL tasks. LEDD also provides a simple Python interface to facilitate the extension and the replacement of data discovery algorithms.