Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a Quality Indicator for Research Data publications and Research Software publications -- A vision from the Helmholtz Association

Wolfgang zu Castell, Doris Dransch, Guido Juckeland, Marcel Meistring, Bernadette Fritzsch, Ronny Gey, Britta Höpfner, Martin Köhler, Christian Meeßen, Hela Mehrtens, Felix Mühlbauer, Sirko Schindler, Thomas Schnicke, Roland Bertelmann

TL;DR

This paper proposes a multi-dimensional quality indicator for research data and software publications within the Helmholtz Association, combining FAIR-based criteria with a COBIT-inspired maturity model and radar-plot visualization to yield a single, comparable score. It defines data and software quality through six data-oriented (Publishing, Openness, Curation, Metadata, External View) and six software-oriented (Findable, Accessible, Interoperable, Reusable, Scientific basis, Technical basis) dimensions, each populated by structured attributes whose maturity is scored and aggregated. The framework emphasizes automation and standardized metadata, while allowing domain-specific external assessments to inform domain-relevant quality checks. The authors present a living, discussion-driven approach intended to foster transparency, openness, and professionalization of data/software publication practices, with pilot implementations and community input guiding future refinements.

Abstract

Research data and software are widely accepted as an outcome of scientific work. However, in comparison to text-based publications, there is not yet an established process to assess and evaluate quality of research data and research software publications. This paper presents an attempt to fill this gap. Initiated by the Working Group Open Science of the Helmholtz Association the Task Group Helmholtz Quality Indicators for Data and Software Publications currently develops a quality indicator for research data and research software publications to be used within the Association. This report summarizes the vision of the group of what all contributes to such an indicator. The proposed approach relies on generic well-established concepts for quality criteria, such as the FAIR Principles and the COBIT Maturity Model. It does - on purpose - not limit itself to technical implementation possibilities to avoid using an existing metric for a new purpose. The intention of this paper is to share the current state for further discussion with all stakeholders, particularly with other groups also working on similar metrics but also with entities that use the metrics.

Towards a Quality Indicator for Research Data publications and Research Software publications -- A vision from the Helmholtz Association

TL;DR

This paper proposes a multi-dimensional quality indicator for research data and software publications within the Helmholtz Association, combining FAIR-based criteria with a COBIT-inspired maturity model and radar-plot visualization to yield a single, comparable score. It defines data and software quality through six data-oriented (Publishing, Openness, Curation, Metadata, External View) and six software-oriented (Findable, Accessible, Interoperable, Reusable, Scientific basis, Technical basis) dimensions, each populated by structured attributes whose maturity is scored and aggregated. The framework emphasizes automation and standardized metadata, while allowing domain-specific external assessments to inform domain-relevant quality checks. The authors present a living, discussion-driven approach intended to foster transparency, openness, and professionalization of data/software publication practices, with pilot implementations and community input guiding future refinements.

Abstract

Research data and software are widely accepted as an outcome of scientific work. However, in comparison to text-based publications, there is not yet an established process to assess and evaluate quality of research data and research software publications. This paper presents an attempt to fill this gap. Initiated by the Working Group Open Science of the Helmholtz Association the Task Group Helmholtz Quality Indicators for Data and Software Publications currently develops a quality indicator for research data and research software publications to be used within the Association. This report summarizes the vision of the group of what all contributes to such an indicator. The proposed approach relies on generic well-established concepts for quality criteria, such as the FAIR Principles and the COBIT Maturity Model. It does - on purpose - not limit itself to technical implementation possibilities to avoid using an existing metric for a new purpose. The intention of this paper is to share the current state for further discussion with all stakeholders, particularly with other groups also working on similar metrics but also with entities that use the metrics.
Paper Structure (51 sections, 1 figure)

This paper contains 51 sections, 1 figure.

Table of Contents

  1. Background and objective
  2. Terminology
  3. Methodical framework – quality criteria and quality quantification
  4. The COBIT Maturity Model
  5. Weighted summation and radar plot
  6. The Quality Indicator for research data
  7. Dimensions of quality for research data assessment
  8. Quality assessment of the dimensions
  9. Dimension Publishing
  10. Published with Identifier
  11. Published via a Repository or Collection, that is indexed in a Meta-Repository (e.g. re3data)
  12. Published with Information on Access to the Data
  13. Dimension Openness
  14. General Degree of Openness
  15. Primary Data Formats
  16. ...and 36 more sections

Figures (1)

  • Figure 1: Example of a fictitious assessment for two research software or data publications showing how the radar plot can visualize the fulfillment of some criteria at higher quality levels than others. Maturity levels are ordered from center (0, non-existent) to outside (5, optimized). The minimum requirements and maximum scores might also differ for each dimension.