Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A City upon a Hill: Casting Light on a Real Experimental Process

Efraín R. Fonseca C., Marta López-Fernández, Oscar Dieste, Natalia Juristo

TL;DR

This ethnographic study investigates how empirical software engineering researchers actually conduct experiments, revealing a landscape broader than textbook accounts. Through long-term observation, literature review, replication of a baseline RGUS experiment, and tacit-knowledge mining, the authors develop conceptual and process models that capture roles, activities, and the workflow of SE experimentation. Key findings show diverse activities, distinct roles, and substantial tacit knowledge shaping practice, with implications for replication and education. The work argues for embracing family-specific practices and tacit knowledge to improve replicability and training in SE research.

Abstract

Context: The overall scientific community is proposing measures to improve the reproducibility and replicability of experiments. Reproducibility is relatively easy to achieve. However, replicability is considerably more complex in both the sciences and Empirical Software Engineering (ESE). Several strategies, e.g., replication packages and families of experiments, have been proposed to improve replication in ESE, with limited success. We wonder whether the failures are due to some mismatch, i.e., the researchers' needs are not satisfied by the proposed replication procedures. Objectives: Find out how experimental researchers conduct \textit{experiments in practice}. Methods: We carried out an ethnography study within a SE Research Group. Our main activity was to observe/approach the experimental researchers in their day-to-day settings for two years. Their preferred literature and experimental materials were studied. We used individual and group interviews to gain understanding and examine unclear topics in-depth. Results: We have created conceptual and process models that represent how experimentation is really conducted in the Research Group. Models fit the community's procedures and terminology at a high level, but they become particular in their minute details. Conclusion: The actual experimental process differs from textbooks in several points, namely: (1) Number and diversity of activities, (2) existence of different roles, (3) the granularity of the concepts used by the roles, and (4) the viewpoints that different sub-areas or families of experiments have about the overall process.

A City upon a Hill: Casting Light on a Real Experimental Process

TL;DR

This ethnographic study investigates how empirical software engineering researchers actually conduct experiments, revealing a landscape broader than textbook accounts. Through long-term observation, literature review, replication of a baseline RGUS experiment, and tacit-knowledge mining, the authors develop conceptual and process models that capture roles, activities, and the workflow of SE experimentation. Key findings show diverse activities, distinct roles, and substantial tacit knowledge shaping practice, with implications for replication and education. The work argues for embracing family-specific practices and tacit knowledge to improve replicability and training in SE research.

Abstract

Context: The overall scientific community is proposing measures to improve the reproducibility and replicability of experiments. Reproducibility is relatively easy to achieve. However, replicability is considerably more complex in both the sciences and Empirical Software Engineering (ESE). Several strategies, e.g., replication packages and families of experiments, have been proposed to improve replication in ESE, with limited success. We wonder whether the failures are due to some mismatch, i.e., the researchers' needs are not satisfied by the proposed replication procedures. Objectives: Find out how experimental researchers conduct \textit{experiments in practice}. Methods: We carried out an ethnography study within a SE Research Group. Our main activity was to observe/approach the experimental researchers in their day-to-day settings for two years. Their preferred literature and experimental materials were studied. We used individual and group interviews to gain understanding and examine unclear topics in-depth. Results: We have created conceptual and process models that represent how experimentation is really conducted in the Research Group. Models fit the community's procedures and terminology at a high level, but they become particular in their minute details. Conclusion: The actual experimental process differs from textbooks in several points, namely: (1) Number and diversity of activities, (2) existence of different roles, (3) the granularity of the concepts used by the roles, and (4) the viewpoints that different sub-areas or families of experiments have about the overall process.

Paper Structure

This paper contains 42 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Research Method
  3. Context selection
  4. Data collection procedures
  5. Data analysis procedure
  6. Ethnographic study
  7. First Approximation
  8. Analysis of the relevant literature
  9. Review of experimental material
  10. Diving into the SE experimental process
  11. Experiment selection
  12. Replication planning
  13. Conduction
  14. Analysis and reporting
  15. Assessment
  16. ...and 27 more sections

Figures (3)

  • Figure 1: Example of a sketch made by experimenters.
  • Figure 2: Extract from the experimental activities carried out by the RGUS experimenters (intermediate version). The evolution regarding Fig. \ref{['fig-proceso-exp-boseto']} is observable in terms of the number of activities performed and transparency in the workflow.
  • Figure 3: Relationships between the experimental process workflow (top) and the experimental process model (down).