Metadata-based Data Exploration with Retrieval-Augmented Generation for Large Language Models

TL;DR

The results demonstrate that RAG can enhance the selection of relevant datasets, particularly from different categories, when compared to conventional metadata approaches, however, performance varied across tasks and models, which confirms the significance of selecting appropriate techniques based on specific use cases.

Abstract

Developing the capacity to effectively search for requisite datasets is an urgent requirement to assist data users in identifying relevant datasets considering the very limited available metadata. For this challenge, the utilization of third-party data is emerging as a valuable source for improvement. Our research introduces a new architecture for data exploration which employs a form of Retrieval-Augmented Generation (RAG) to enhance metadata-based data discovery. The system integrates large language models (LLMs) with external vector databases to identify semantic relationships among diverse types of datasets. The proposed framework offers a new method for evaluating semantic similarity among heterogeneous data sources and for improving data exploration. Our study includes experimental results on four critical tasks: 1) recommending similar datasets, 2) suggesting combinable datasets, 3) estimating tags, and 4) predicting variables. Our results demonstrate that RAG can enhance the selection of relevant datasets, particularly from different categories, when compared to conventional metadata approaches. However, performance varied across tasks and models, which confirms the significance of selecting appropriate techniques based on specific use cases. The findings suggest that this approach holds promise for addressing challenges in data exploration and discovery, although further refinement is necessary for estimation tasks.

Figures (5)

• Figure 1: The proposed RAG system architecture for data exploration.
• Figure 2: Output results of (a) similar dataset recommendation (Task 1) with four language models and three metadata input types and (b) combinable dataset recommendation (Task 2). The both bar graphs depict the sources of recommended datasets classified into three groups: “datasets from different categories (red),” “datasets from the same category (blue),” and “datasets generated by LLM (lightgreen).” The first two groups consist of actual datasets in HDX, while the third category comprises fictional datasets generated by LLM.
• Figure 3: Comparison of mean variable/description similarities of recommended datasets before and after via LLM in Task 1. The bar graphs depict the sources of recommended datasets classified into two groups: “datasets from different categories (red),” and “datasets from the same category (blue).” The error bars show the standard deviation.
• Figure 4: Comparison of mean variable/description similarities of recommended datasets before and after via LLM in Task 2. The bar graphs depict the sources of recommended datasets classified into two groups: “datasets from different categories (red),” and “datasets from the same category (blues).” The error bars show the standard deviation.
• Figure 5: Mean F1 scores in (a) Task 3, and (b) Task 4. The bar graphs on the left of each model (orange) are the scores for the tags/variables contained in the metadata output from the vector DBs, and the ones on the right (green) are the scores for those selected via LLM.