Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Artifact Sharing for Information Retrieval Research

Sean MacAvaney

TL;DR

The paper tackles the lack of a standard for sharing IR artifacts beyond code and models by introducing a flexible, interoperable artifact-sharing system built around a simple Artifact Serialization File, pluggable Artifact Classes, and multiple Artifact Hosts. Integrated with PyTerrier, the system supports numerous artifact types (indexes, caches, datasets) across seven hosting platforms, with metadata and adapters to ensure correct loading and use. Key contributions include a scalable serialization format with type/format metadata, a registry of 14+ Artifact Classes across languages, and practical demonstrations showing seamless load/share workflows, including cross-institution collaborations. The work enhances reproducibility, accessibility, and sustainability of IR research by enabling broad, low-overhead artifact sharing with immediate usability.

Abstract

Sharing artifacts -- such as trained models, pre-built indexes, and the code to use them -- aids in reproducibility efforts by allowing researchers to validate intermediate steps and improves the sustainability of research by allowing multiple groups to build off one another's prior computational work. Although there are de facto consensuses on how to share research code (through a git repository linked to from publications) and trained models (via HuggingFace Hub), there is no consensus for other types of artifacts, such as built indexes. Given the practical utility of using shared indexes, researchers have resorted to self-hosting these resources or performing ad hoc file transfers upon request, ultimately limiting the artifacts' discoverability and reuse. This demonstration introduces a flexible and interoperable way to share artifacts for Information Retrieval research, improving both their accessibility and usability.

Artifact Sharing for Information Retrieval Research

TL;DR

The paper tackles the lack of a standard for sharing IR artifacts beyond code and models by introducing a flexible, interoperable artifact-sharing system built around a simple Artifact Serialization File, pluggable Artifact Classes, and multiple Artifact Hosts. Integrated with PyTerrier, the system supports numerous artifact types (indexes, caches, datasets) across seven hosting platforms, with metadata and adapters to ensure correct loading and use. Key contributions include a scalable serialization format with type/format metadata, a registry of 14+ Artifact Classes across languages, and practical demonstrations showing seamless load/share workflows, including cross-institution collaborations. The work enhances reproducibility, accessibility, and sustainability of IR research by enabling broad, low-overhead artifact sharing with immediate usability.

Abstract

Sharing artifacts -- such as trained models, pre-built indexes, and the code to use them -- aids in reproducibility efforts by allowing researchers to validate intermediate steps and improves the sustainability of research by allowing multiple groups to build off one another's prior computational work. Although there are de facto consensuses on how to share research code (through a git repository linked to from publications) and trained models (via HuggingFace Hub), there is no consensus for other types of artifacts, such as built indexes. Given the practical utility of using shared indexes, researchers have resorted to self-hosting these resources or performing ad hoc file transfers upon request, ultimately limiting the artifacts' discoverability and reuse. This demonstration introduces a flexible and interoperable way to share artifacts for Information Retrieval research, improving both their accessibility and usability.
Paper Structure (7 sections, 1 figure, 2 tables)

This paper contains 7 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. System at a Glance
  3. Technical Details
  4. Artifact Serialization File and Metadata
  5. Artifact Classes
  6. Artifact Hosts
  7. Conclusion

Figures (1)

  • Figure 1: An overview of the distributed design for sharing IR artifacts. Artifacts (instances of Artifact Classes) can be serialized and shared to an Artifact Host via a Artifact Serialization File. Hosted artifacts can then be downloaded and loaded as the original Artifact Class.