Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guiding Principles for Using Mixed Methods Research in Software Engineering

Margaret-Anne Storey, Rashina Hoda, Alessandra Maciel Paz Milani, Maria Teresa Baldassarre

TL;DR

Guiding Principles for Using Mixed Methods Research in Software Engineering defines mixed methods research (MMR) in SE and argues for principled, rigorous, and ethical designs. It articulates four core guiding principles—methodological rationale, novel integrated insights (meta-inferences), procedural rigor, and ethical research—and outlines the design landscape (design properties, types of integration, and common designs). Through four fictional SE design scenarios mapped to Exploratory Sequential, Explanatory Sequential, Convergent Parallel, and Embedded designs, the paper demonstrates how to trade off design choices and avoid antipatterns. It also discusses synergies with other methods and provides a call to action for the SE community to adopt open, principled, and human-centered MMR practices, along with an appendix glossary of terms.

Abstract

Mixed methods research is often used in software engineering, but researchers outside of the social or human sciences often lack experience when using these designs. This paper provides guiding principles and advice on how to design mixed method research, and to encourage the intentional, rigorous, and innovative application of mixed methods in software engineering. It also presents key properties of core mixed method research designs. Through a number of fictitious but recognizable software engineering research scenarios, we showcase how to choose suitable designs and consider the inevitable trade-offs any design choice leads to. We describe several antipatterns that illustrate what to avoid in mixed method research, and when mixed method research should be considered over other approaches.

Guiding Principles for Using Mixed Methods Research in Software Engineering

TL;DR

Guiding Principles for Using Mixed Methods Research in Software Engineering defines mixed methods research (MMR) in SE and argues for principled, rigorous, and ethical designs. It articulates four core guiding principles—methodological rationale, novel integrated insights (meta-inferences), procedural rigor, and ethical research—and outlines the design landscape (design properties, types of integration, and common designs). Through four fictional SE design scenarios mapped to Exploratory Sequential, Explanatory Sequential, Convergent Parallel, and Embedded designs, the paper demonstrates how to trade off design choices and avoid antipatterns. It also discusses synergies with other methods and provides a call to action for the SE community to adopt open, principled, and human-centered MMR practices, along with an appendix glossary of terms.

Abstract

Mixed methods research is often used in software engineering, but researchers outside of the social or human sciences often lack experience when using these designs. This paper provides guiding principles and advice on how to design mixed method research, and to encourage the intentional, rigorous, and innovative application of mixed methods in software engineering. It also presents key properties of core mixed method research designs. Through a number of fictitious but recognizable software engineering research scenarios, we showcase how to choose suitable designs and consider the inevitable trade-offs any design choice leads to. We describe several antipatterns that illustrate what to avoid in mixed method research, and when mixed method research should be considered over other approaches.
Paper Structure (50 sections, 7 figures, 1 table)

This paper contains 50 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. A Brief Overview of Mixed Methods Research
  3. Principles to Guide Mixed Method Research
  4. Methodological Rationale
  5. Novel Integrated Insights
  6. Procedural Rigor
  7. Ethical Research
  8. A Landscape of Mixed Methods Design
  9. Design Properties
  10. Types of Integration
  11. Research Designs
  12. Mixed Method Research Design Scenarios
  13. Ali's Study: Investigating the impact of a new security tool on developer productivity and developer experience at a large software company.
  14. Methodological Rationale:
  15. Novelty of Integrated Insights:
  16. ...and 35 more sections

Figures (7)

  • Figure 1: Interconnection of the core components for MMR: collecting, analyzing and interpreting (1) qualitative and (2) quantitative data to address a research question; (3) integrating the data (merging, connecting, and building) in a research design and set of procedures; and (4) drawing meta-inferences that add new insight beyond the quaL and quaN data. Adapted from Creswell and Plano Clark poth_sage_2023.
  • Figure 2: Four Guiding Principles for Mixed Method Research in Software Engineering.
  • Figure 3: Ali follows an exploratory sequential design, studying the impact of a security tool that enforces secure development practices. The unit of analysis are the practices informed by the developers using the tool across a company.
  • Figure 4: Sam follows an explanatory sequential design to study the adoption of a generative AI reviewing tool on code quality. The unit of analysis is telemetry process metrics in (1), while in (2) it is perception of the process informed by developers.
  • Figure 5: Vicki follows a planned convergent parallel design to study the adoption of a new onboarding process in an open source project.
  • ...and 2 more figures