What Do Contribution Guidelines Say About Software Testing?

TL;DR

This paper investigates how software testing is addressed in open-source contribution guidelines by analyzing 200 Python and JavaScript projects. Using a two-question framework, it quantifies the prevalence of test documentation and characterizes its content, revealing that 78% of projects include test guidance, predominantly about running tests and unit tests, with relatively sparse coverage of integration, end-to-end testing, coverage, and mocking. The study highlights a bias toward how-to-run guidance over how-to-write tests and stresses the need for clearer, more comprehensive testing guidance to aid contributors and improve contribution quality. The findings offer practical implications for maintainers and researchers, including guidelines for improving test-oriented documentation and opportunities for tooling to detect content gaps in guidelines.

Abstract

Software testing plays a crucial role in the contribution process of open-source projects. For example, contributions introducing new features are expected to include tests, and contributions with tests are more likely to be accepted. Although most real-world projects require contributors to write tests, the specific testing practices communicated to contributors remain unclear. In this paper, we present an empirical study to understand better how software testing is approached in contribution guidelines. We analyze the guidelines of 200 Python and JavaScript open-source software projects. We find that 78\% of the projects include some form of test documentation for contributors. Test documentation is located in multiple sources, including \texttt{CONTRIBUTING} files (58\%), external documentation (24\%), and \texttt{README} files (8\%). Furthermore, test documentation commonly explains how to run tests (83.5\%), but less often provides guidance on how to write tests (37\%). It frequently covers unit tests (71\%), but rarely addresses integration (20.5\%) and end-to-end tests (15.5\%). Other key testing aspects are also less frequently discussed: test coverage (25.5\%) and mocking (9.5\%). We conclude by discussing implications and future research.