Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenges, Adaptations, and Fringe Benefits of Conducting Software Engineering Research with Human Participants during the COVID-19 Pandemic

Anuradha Madugalla, Tanjila Kanij, Rashina Hoda, Dulaji Hidellaarachchi, Aastha Pant, Samia Ferdousi, John Grundy

TL;DR

The paper addresses how the COVID-19 pandemic disrupted software engineering research that involves human participants. It adopts a sequential mixed-methods approach, collecting a global survey (n=89) and nine in-depth interviews, and analyzes data with socio-technical grounded theory (STGT4DA) and descriptive statistics. The study identifies key challenges in research design, recruitment, and data collection, documents effective adaptations, and highlights fringe benefits such as increased geographic diversity and reduced costs, while offering practical guidelines for conducting remote and hybrid studies. The findings inform best practices for future SE research in hybrid environments where face-to-face engagement is limited or unnecessary, with implications for researchers, reviewers, and program chairs in how to design, recruit for, and execute human-participant studies.

Abstract

The COVID-19 pandemic changed the way we live, work and the way we conduct research. With the restrictions of lockdowns and social distancing, various impacts were experienced by many software engineering researchers, especially whose studies depend on human participants. We conducted a mixed methods study to understand the extent of this impact. Through a detailed survey with 89 software engineering researchers working with human participants around the world and a further nine follow-up interviews, we identified the key challenges faced, the adaptations made, and the surprising fringe benefits of conducting research involving human participants during the pandemic. Our findings also revealed that in retrospect, many researchers did not wish to revert to the old ways of conducting human-oriented research. Based on our analysis and insights, we share recommendations on how to conduct remote studies with human participants effectively in an increasingly hybrid world when face-to-face engagement is not possible or where remote participation is preferred.

Challenges, Adaptations, and Fringe Benefits of Conducting Software Engineering Research with Human Participants during the COVID-19 Pandemic

TL;DR

The paper addresses how the COVID-19 pandemic disrupted software engineering research that involves human participants. It adopts a sequential mixed-methods approach, collecting a global survey (n=89) and nine in-depth interviews, and analyzes data with socio-technical grounded theory (STGT4DA) and descriptive statistics. The study identifies key challenges in research design, recruitment, and data collection, documents effective adaptations, and highlights fringe benefits such as increased geographic diversity and reduced costs, while offering practical guidelines for conducting remote and hybrid studies. The findings inform best practices for future SE research in hybrid environments where face-to-face engagement is limited or unnecessary, with implications for researchers, reviewers, and program chairs in how to design, recruit for, and execute human-participant studies.

Abstract

The COVID-19 pandemic changed the way we live, work and the way we conduct research. With the restrictions of lockdowns and social distancing, various impacts were experienced by many software engineering researchers, especially whose studies depend on human participants. We conducted a mixed methods study to understand the extent of this impact. Through a detailed survey with 89 software engineering researchers working with human participants around the world and a further nine follow-up interviews, we identified the key challenges faced, the adaptations made, and the surprising fringe benefits of conducting research involving human participants during the pandemic. Our findings also revealed that in retrospect, many researchers did not wish to revert to the old ways of conducting human-oriented research. Based on our analysis and insights, we share recommendations on how to conduct remote studies with human participants effectively in an increasingly hybrid world when face-to-face engagement is not possible or where remote participation is preferred.
Paper Structure (28 sections, 4 figures, 3 tables)

This paper contains 28 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Impact of the Pandemic on Research
  4. Impact of the Pandemic on Software Engineering Research
  5. Methodology
  6. Research Design
  7. Recruitment
  8. Data Collection
  9. Data Analysis
  10. Findings
  11. General Findings
  12. Demographics
  13. Number of Studies
  14. Data collection methods and participant groups
  15. Challenges, Adaptations, and Fringe Benefits
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Mixed methods research using socio-technical grounded theory for data analysis (STGT4DA) of qualitative data hoda2021STGT and statistical analysis of quantitative data.
  • Figure 2: Example of applying STGT for data analysis.
  • Figure 3: Example of a memo written as part of the STGT for the data analysis process.
  • Figure 4: Key findings of the study