Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Evaluation Framework for the FAIR Assessment tools in Open Science

Payel Patra, Daniele Di Pompeo, Antinisca Di Marco

TL;DR

This work tackles the lack of standardized evaluation for FAIR assessment tools (FAIR-a) used with Open Science Platforms. It develops a bottom-up evaluation framework comprising 19 attributes across four dimensions (functionality, technical, runtime, usability) and applies it to 22 publicly available FAIR-a tools, using SoBigData RI as the reference OSP and MMASH data for validation. The study identifies top-performing tools from a user perspective (e.g., FAIR-Checker, FAIR enough, F-UJI, SATIFYD, FAIR-Aware, FAIR data Self-Assessment Tool) and highlights practical challenges such as data-type compatibility and guidance quality. The contributions offer a structured basis for tool selection, enable reproducible assessments, and point to targeted improvements for developers to advance FAIR practice in open science.

Abstract

Open science represents a transformative research approach essential for enhancing sustainability and impact. Data generation encompasses various methods, from automated processes to human-driven inputs, creating a rich and diverse landscape. Embracing the FAIR principles -- making data and, in general, artifacts (such as code, configurations, documentation, etc) findable, accessible, interoperable, and reusable -- ensures research integrity, transparency, and reproducibility, and researchers enhance the efficiency and efficacy of their endeavors, driving scientific innovation and the advancement of knowledge. Open Science Platforms OSP (i.e., technologies that publish data in a way that they are findable, accessible, interoperable, and reusable) are based on open science guidelines and encourage accessibility, cooperation, and transparency in scientific research. Evaluating OSP will yield sufficient data and artifacts to enable better sharing and arrangement, stimulating more investigation and the development of new platforms. In this paper, we propose an evaluation framework that results from evaluating twenty-two FAIR-a tools assessing the FAIR principles of OSP to identify differences, shortages, and possible efficiency improvements.

An Evaluation Framework for the FAIR Assessment tools in Open Science

TL;DR

This work tackles the lack of standardized evaluation for FAIR assessment tools (FAIR-a) used with Open Science Platforms. It develops a bottom-up evaluation framework comprising 19 attributes across four dimensions (functionality, technical, runtime, usability) and applies it to 22 publicly available FAIR-a tools, using SoBigData RI as the reference OSP and MMASH data for validation. The study identifies top-performing tools from a user perspective (e.g., FAIR-Checker, FAIR enough, F-UJI, SATIFYD, FAIR-Aware, FAIR data Self-Assessment Tool) and highlights practical challenges such as data-type compatibility and guidance quality. The contributions offer a structured basis for tool selection, enable reproducible assessments, and point to targeted improvements for developers to advance FAIR practice in open science.

Abstract

Open science represents a transformative research approach essential for enhancing sustainability and impact. Data generation encompasses various methods, from automated processes to human-driven inputs, creating a rich and diverse landscape. Embracing the FAIR principles -- making data and, in general, artifacts (such as code, configurations, documentation, etc) findable, accessible, interoperable, and reusable -- ensures research integrity, transparency, and reproducibility, and researchers enhance the efficiency and efficacy of their endeavors, driving scientific innovation and the advancement of knowledge. Open Science Platforms OSP (i.e., technologies that publish data in a way that they are findable, accessible, interoperable, and reusable) are based on open science guidelines and encourage accessibility, cooperation, and transparency in scientific research. Evaluating OSP will yield sufficient data and artifacts to enable better sharing and arrangement, stimulating more investigation and the development of new platforms. In this paper, we propose an evaluation framework that results from evaluating twenty-two FAIR-a tools assessing the FAIR principles of OSP to identify differences, shortages, and possible efficiency improvements.

Paper Structure

This paper contains 22 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Research Motivation and Objectives
  3. Research Methodology
  4. Evaluation results
  5. Addressing RQ1: Evaluation Framework Specification
  6. Answer to RQ1:
  7. Addressing RQ2: using the evaluation framework to classify existing FAIR-a Tools
  8. Online Self-Assessment tools evaluation
  9. Assessment of SATIFYD tool:
  10. Online Semi-Automated tools evaluation
  11. Assessment of FAIRshake tool:
  12. Online Automated tools evaluation
  13. F-UJI assessment:
  14. Offline Self-Assessment tools evaluation
  15. Assessment of "Do I-PASS for FAIR" tool:
  16. ...and 7 more sections

Figures (2)

  • Figure 1: FAIR-a tools evaluation methodology
  • Figure 2: This figure represents Evaluation Grade as bars with FAIR-a Tool Names.