FAIRSECO: An Extensible Framework for Impact Measurement of Research Software

TL;DR

FAIRSECO addresses the lack of a unified framework to credit researchers and Research Software Engineers by integrating quality, FAIRness, and impact metrics for research software. It combines tools for FAIRness (howfairis), licensing (Tortellini), and code reuse (SearchSECO) with bibliometric data from OpenAlex and Semantic Scholar to compute two core scores: $S_{quality}$ and $S_{impact}$. The framework employs an extensible architecture with data collection, code analysis, impact calculation, reporting, and artifact generation (including SBOMs and license reports). Through a concrete in-action demonstration and evaluation across RS projects, FAIRSECO highlights its potential to streamline recognition, guide improvements, and quantify RS impact, while outlining future enhancements like broader platform support and NLP-based documentation scoring.

Abstract

The growing usage of research software in the research community has highlighted the need to recognize and acknowledge the contributions made not only by researchers but also by Research Software Engineers. However, the existing methods for crediting research software and Research Software Engineers have proven to be insufficient. In response, we have developed FAIRSECO, an extensible open source framework with the objective of assessing the impact of research software in research through the evaluation of various factors. The FAIRSECO framework addresses two critical information needs: firstly, it provides potential users of research software with metrics related to software quality and FAIRness. Secondly, the framework provides information for those who wish to measure the success of a project by offering impact data. By exploring the quality and impact of research software, our aim is to ensure that Research Software Engineers receive the recognition they deserve for their valuable contributions.

Figures (4)

• Figure 1: FAIRSECO architecture consists of five components: 1) Data collection unit gathers data from different data sources, 2) Impact calculation performs an impact calculation, 3) Code analyzer analyzes the code of the RSE project, 4) Report generates final report related to Quality and Impact, and 5) Artifacts generates two artifacts, such as SBOM, and License results for the RS.
• Figure 2: Overview tab: This screenshot displays a summary of the FAIRSECO report, including the number of citations, $S_{fair}$, matching methods, $S_{quality}$, and several metrics from GitHub. For a more detailed understanding of the $S_{quality}$, please refer to Section \ref{['toolaction']}
• Figure 3: Impact tab: The Impact tab screenshot presents an overview of the RS Mcfly impact. It has been cited a total of 4 times, with two of those citations coming from different fields. Additionally, the screenshot includes a radar graph that illustrates the software's citation frequency across various fields. For a more detailed understanding of this tab, please refer to Section \ref{['toolaction']}.
• Figure 4: Quality Score tab: This screenshot shows an overview of the generated FAIRSECO $S_{quality}$. For a complete explanation of the $S_{quality}$, see Section \ref{['toolaction']}