SARA: A Collection of Sensitivity-Aware Relevance Assessments

TL;DR

This work addresses the need for publicly accessible resources to develop sensitivity-aware search by extending the Enron email collection with Sensitivity-Aware Relevance Assessments (SARA). Using Latent Dirichlet Allocation, the authors identify 50 topics, generate short information-needs descriptions, and collect crowdsourced queries and relevance judgments (11,471 total) to create a rich testbed that pairs information needs with sensitivity labels. They demonstrate the utility of the dataset through baseline post-filtering sensitivity classifiers (SVM and LR) and BM25-based retrieval, showing improved metrics when sensitive documents are filtered, and provide the resources openly via GitHub and ir_datasets under CC BY-NC 4.0. The dataset enables evaluation of sensitivity-aware search models and facilitates future research into generalizable handling of sensitive information across diverse domains.

Abstract

Large archival collections, such as email or government documents, must be manually reviewed to identify any sensitive information before the collection can be released publicly. Sensitivity classification has received a lot of attention in the literature. However, more recently, there has been increasing interest in developing sensitivity-aware search engines that can provide users with relevant search results, while ensuring that no sensitive documents are returned to the user. Sensitivity-aware search would mitigate the need for a manual sensitivity review prior to collections being made available publicly. To develop such systems, there is a need for test collections that contain relevance assessments for a set of information needs as well as ground-truth labels for a variety of sensitivity categories. The well-known Enron email collection contains a classification ground-truth that can be used to represent sensitive information, e.g., the Purely Personal and Personal but in Professional Context categories can be used to represent sensitive personal information. However, the existing Enron collection does not contain a set of information needs and relevance assessments. In this work, we present a collection of fifty information needs (topics) with crowdsourced query formulations (3 per topic) and relevance assessments (11,471 in total) for the Enron collection (mean number of relevant documents per topic = 11, variance = 34.7). The developed information needs, queries and relevance judgements are available on GitHub and will be available along with the existing Enron collection through the popular ir_datasets library. Our proposed collection results in the first freely available test collection for developing sensitivity-aware search systems.

Figures (5)

• Figure 1: Enron Classification Categories.
• Figure 2: Distribution of classification labels across the subcategories of the Coarse Genre category
• Figure 3: Two example information needs and queries gathered from the crowdsourcing tasks.
• Figure 4: The system used for the relevance assessments crowdworking task
• Figure 5: The number of relevant documents per information need.