A Disruptive Research Playbook for Studying Disruptive Innovations

TL;DR

The paper addresses the need for a socio-technical approach to studying disruptive software engineering technologies. It introduces a Disruptive Research Playbook, grounded in McLuhan's tetrad and McGrath's triadic domains, to guide researchers from problem framing to empirical strategies and includes a GPT-based companion, MyResearchPlaybook. Through retrospective analysis of Stack Overflow and forward-looking applications to generative AI in program synthesis and AR/VR in team collaboration, the work demonstrates how the playbook yields socially relevant research questions and methods. The main contributions are the structured playbook, the integration of theory with practice, and the MyResearchPlaybook tool to aid brainstorming. The practical significance lies in steering software engineering research toward human-centered, long-term, socio-technical impacts of disruptive technologies.

Abstract

As researchers, we are now witnessing a fundamental change in our technologically-enabled world due to the advent and diffusion of highly disruptive technologies such as generative AI, Augmented Reality (AR) and Virtual Reality (VR). In particular, software engineering has been profoundly affected by the transformative power of disruptive innovations for decades, with a significant impact of technical advancements on social dynamics due to its the socio-technical nature. In this paper, we reflect on the importance of formulating and addressing research in software engineering through a socio-technical lens, thus ensuring a holistic understanding of the complex phenomena in this field. We propose a research playbook with the goal of providing a guide to formulate compelling and socially relevant research questions and to identify the appropriate research strategies for empirical investigations, with an eye on the long-term implications of technologies or their use. We showcase how to apply the research playbook. Firstly, we show how it can be used retrospectively to reflect on a prior disruptive technology, Stack Overflow, and its impact on software development. Secondly, we show it can be used to question the impact of two current disruptive technologies: AI and AR/VR. Finally, we introduce a specialized GPT model to support the researcher in framing future investigations. We conclude by discussing the broader implications of adopting the playbook for both researchers and practitioners in software engineering and beyond.

Figures (6)

• Figure 1: The McLuhan's Tetrad showcasing the four laws that every new medium follows.
• Figure 2: McLuhan's Tetrad applied to Stack Overflow as one example of a disruptive innovation in SE. For each law, we reflect on how Stack Overflow has had an impact on software developers and their practices. Impacts noted with an asterisk were suggested by ChatGPT 4 on Oct 16, 2023.
• Figure 3: Methodological Domain: Empirical and non-empirical research strategies. Empirical strategies (on the right) are annotated with the quality criteria they have potential to maximize, and which strategies involve direct involvement of human participants (all except data strategies) with the blue person icon. The quality criteria shown on the outside of the empirical circle reveals the inevitable tradeoffs made when a particular strategy is selected.
• Figure 4: A research playbook for evaluating the impact of disruptive technologies. Step 1 builds on McLuhan's tetrad to brainstorm broad questions. Step 2 identifies phenomena and ideas to study. Step 3 creates a matrix of research questions. Step 4 suggests research strategies that focus on human aspects. The steps may be followed iteratively, refining phenomena, ideas and questions in terms of their potential research impact.
• Figure 5: Step 4: This figure shows how we can use this part of the playbook to reflect back on the choice of research strategies to select research studies. Three of these strategies as mentioned directly involve human subjects, but we noted that most studies on Stack Overflow relied on data strategies (many trading precision of data and generalizability for control over human subjects).