Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DSEBench: A Test Collection for Explainable Dataset Search with Examples

Qing Shi, Jing He, Qiaosheng Chen, Gong Cheng

TL;DR

The paper defines Dataset Search with Examples (DSE) and Explainable DSE (ExDSE) to address scenarios where users use a query plus example target datasets. It presents DSEBench as the first public benchmark with dataset-level and field-level annotations, augmented by synthetic training data generated via a large language model. A broad suite of retrieval, reranking, and explainability baselines is evaluated, revealing that dense representations and fine-tuning boost performance and that LLM-based annotations are effective for training, while explainability methods vary in success. The benchmark paves the way for rigorous evaluation and development of combined search and explanation techniques across heterogeneous datasets, with ongoing plans for expansion and community challenges.

Abstract

Dataset search is a well-established task in the Semantic Web and information retrieval research. Current approaches retrieve datasets either based on keyword queries or by identifying datasets similar to a given target dataset. These paradigms fail when the information need involves both keywords and target datasets. To address this gap, we investigate a generalized task, Dataset Search with Examples (DSE), and extend it to Explainable DSE (ExDSE), which further requires identifying relevant fields of the retrieved datasets. We construct DSEBench, the first test collection that provides high-quality dataset-level and field-level annotations to support the evaluation of DSE and ExDSE, respectively. In addition, we employ a large language model to generate extensive annotations for training purposes. We establish comprehensive baselines on DSEBench by adapting and evaluating a variety of lexical, dense, and LLM-based retrieval, reranking, and explanation methods.

DSEBench: A Test Collection for Explainable Dataset Search with Examples

TL;DR

The paper defines Dataset Search with Examples (DSE) and Explainable DSE (ExDSE) to address scenarios where users use a query plus example target datasets. It presents DSEBench as the first public benchmark with dataset-level and field-level annotations, augmented by synthetic training data generated via a large language model. A broad suite of retrieval, reranking, and explainability baselines is evaluated, revealing that dense representations and fine-tuning boost performance and that LLM-based annotations are effective for training, while explainability methods vary in success. The benchmark paves the way for rigorous evaluation and development of combined search and explanation techniques across heterogeneous datasets, with ongoing plans for expansion and community challenges.

Abstract

Dataset search is a well-established task in the Semantic Web and information retrieval research. Current approaches retrieve datasets either based on keyword queries or by identifying datasets similar to a given target dataset. These paradigms fail when the information need involves both keywords and target datasets. To address this gap, we investigate a generalized task, Dataset Search with Examples (DSE), and extend it to Explainable DSE (ExDSE), which further requires identifying relevant fields of the retrieved datasets. We construct DSEBench, the first test collection that provides high-quality dataset-level and field-level annotations to support the evaluation of DSE and ExDSE, respectively. In addition, we employ a large language model to generate extensive annotations for training purposes. We establish comprehensive baselines on DSEBench by adapting and evaluating a variety of lexical, dense, and LLM-based retrieval, reranking, and explanation methods.
Paper Structure (20 sections, 4 figures, 7 tables)

This paper contains 20 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Search Benchmarks
  4. Explainable Dataset Search
  5. Relevance Feedback
  6. Construction of DSEBench
  7. Dataset Preprocessing
  8. Test and Training Cases
  9. Pooling
  10. Human Annotation
  11. LLM Annotation
  12. Evaluation of DSE
  13. Training, Validation, and Test Sets
  14. Evaluation Metrics
  15. Benchmarking Retrieval Models
  16. ...and 5 more sections

Figures (4)

  • Figure 1: An example of ExDSE. Left: The user submits a query and a target dataset. Right: The system aims to return positive candidate datasets that exhibit both relevance to the query (highlighted in yellow and green) and similarity to the target dataset (highlighted in magenta, cyan, and gray), as opposed to negative ones that are only relevant to the query or only similar to the target dataset. These judgments are supported by specific metadata and content fields.
  • Figure 2: A screenshot of our tool for human annotation. Left: An input of DSE, including a query and a target dataset. Right: A candidate dataset with annotation options. In this example, the candidate dateset is annotated partially similar to the target dataset. Accordingly, the second column of checkboxes are enabled, and three fields of the candidate dataset are annotated indicators of its similarity. However, the candidate dateset is annotated irrelevant to the query, so the first column of checkboxes representing indicators of relevance are disabled.
  • Figure 3: Inter-annotator agreements on dataset-level judgments for the test cases.
  • Figure 4: Inter-annotator agreements on field-level judgments for the test cases.