RSMM: A Framework to Assess Maturity of Research Software Project

TL;DR

RSMM addresses the lack of a unified framework to assess and mature research software projects. It offers a structured maturity model with 4 focus areas, 17 capabilities, and 79 practices, built from a systematic literature review and expert interviews, and is complemented by the publicly available RSMM-dataset. The model is validated through two case studies (GGIR and ESMValTool), demonstrating its potential to benchmark and guide improvements in software project management, sustainability, community engagement, and adoptability. RSMM aims to support researchers, RSEs, funders, and policy makers in making informed decisions, promoting reproducibility, and driving long-term impact of research software. Future work includes broader case studies and refinement of verification practices to enhance applicability across diverse research contexts.

Abstract

The organizations and researchers producing research software face a common problem of making their software sustainable beyond funding provided by a single research project. This is addressed by research software engineers through building communities around their software, providing appropriate licensing, creating reliable and reproducible research software, making it sustainable and impactful, promoting, and ensuring that the research software is easy to adopt in research workflows, etc. As a result, numerous practices and guidelines exist to enhance research software quality, reusability, and sustainability. However, there is a lack of a unified framework to systematically integrate these practices and help organizations and research software developers refine their development and management processes. Our paper aims at bridging this gap by introducing a novel framework: RSMM. It is designed through systematic literature review and insights from interviews with research software project experts. In short, RSMM offers a structured pathway for evaluating and refining research software project management by categorizing 79 best practices into 17 capabilities across 4 focus areas. From assessing code quality and security to measuring impact, sustainability, and reproducibility, the model provides a complete evaluation of a research software project maturity. With RSMM, individuals as well as organizations involved in research software development gain a systematic approach to tackling various research software engineering challenges. By utilizing RSMM as a comprehensive checklist, organizations can systematically evaluate and refine their project management practices and organizational structure.

Figures (6)

• Figure 1: Design phases of RSMM: The steps involved in developing a RSMM v1.0: In the Scope, Design, and Populate phases, we conducted a systematic literature review to collect and explore academic and grey literature, resulting in the creation of RSMM v0.1. In the Test phase, we included Expert Interviews and Expert Confirmations. Each stage in this phase leads to the evolution of RSMM, also producing intermediate versions v0.2 and v0.3 (available upon request from authors). Case studies are conducted in the Deploy phase to validate the model applicability.
• Figure 2: RSMM v0.1 (not the final model): The initial model includes 4 focus areas, 18 capabilities, and 61 practices. We included the practices collected through the systematic literature review from both academic and grey literature in the model, with maturity levels from 1 to 7. These practices are not placed based on their maturity.
• Figure 3: Maturity Model template by experts: The screenshot depicts 3 experts' ranking of practices within the Requirements capability, illustrating their placements in the RSMM v0.1a expert's dashboard.
• Figure 4: RSMM Focus areas and capabilities: The 4 focus areas and 17 capabilities of the RSMM v1.0 are shown in the figure.
• Figure 5: RSMM v1.0: The updated and final version of RSMM. It includes 4 focus areas, 17 capabilities, and 79 practices. These practices are placed between maturity levels 1 to 10. The first column of the table includes the capability codes.