Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gamifying Testing in IntelliJ: A Replicability Study

Philipp Straubinger, Tommaso Fulcini, Giacomo Garaccione, Luca Ardito, Gordon Fraser

TL;DR

This study replicates the IntelliGame gamification plugin in the TypeScript/JavaScript ecosystem with 174 participants to examine its effects on testing behavior, test suite quality, code functionality, and developer experience. Using a two-group controlled design and a Date Fns task, the authors find that IntelliGame increases test creation, executions, and tool usage, while improving eventual functionality, but does not produce significant gains in coverage or mutation scores and can elevate test smells and perceived time pressure. Correlations between achievement levels and testing metrics suggest that richer achievement progress aligns with more active testing and modest quality improvements. The results generally support the replicability of gamified testing benefits while highlighting nuanced effects on quality and user experience, guiding future refinement of achievement design and longer-term studies. Overall, the work extends prior findings to a new language context and informs practical deployment of IDE-based gamification for software testing education and professional practice.

Abstract

Gamification is an emerging technique to enhance motivation and performance in traditionally unengaging tasks like software testing. Previous studies have indicated that gamified systems have the potential to improve software testing processes by providing testers with achievements and feedback. However, further evidence of these benefits across different environments, programming languages, and participant groups is required. This paper aims to replicate and validate the effects of IntelliGame, a gamification plugin for IntelliJ IDEA to engage developers in writing and executing tests. The objective is to generalize the benefits observed in earlier studies to new contexts, i.e., the TypeScript programming language and a larger participant pool. The replicability study consists of a controlled experiment with 174 participants, divided into two groups: one using IntelliGame and one with no gamification plugin. The study employed a two-group experimental design to compare testing behavior, coverage, mutation scores, and participant feedback between the groups. Data was collected through test metrics and participant surveys, and statistical analysis was performed to determine the statistical significance. Participants using IntelliGame showed higher engagement and productivity in testing practices than the control group, evidenced by the creation of more tests, increased frequency of executions, and enhanced utilization of testing tools. This ultimately led to better code implementations, highlighting the effectiveness of gamification in improving functional outcomes and motivating users in their testing endeavors. The replication study confirms that gamification, through IntelliGame, positively impacts software testing behavior and developer engagement in coding tasks.

Gamifying Testing in IntelliJ: A Replicability Study

TL;DR

This study replicates the IntelliGame gamification plugin in the TypeScript/JavaScript ecosystem with 174 participants to examine its effects on testing behavior, test suite quality, code functionality, and developer experience. Using a two-group controlled design and a Date Fns task, the authors find that IntelliGame increases test creation, executions, and tool usage, while improving eventual functionality, but does not produce significant gains in coverage or mutation scores and can elevate test smells and perceived time pressure. Correlations between achievement levels and testing metrics suggest that richer achievement progress aligns with more active testing and modest quality improvements. The results generally support the replicability of gamified testing benefits while highlighting nuanced effects on quality and user experience, guiding future refinement of achievement design and longer-term studies. Overall, the work extends prior findings to a new language context and informs practical deployment of IDE-based gamification for software testing education and professional practice.

Abstract

Gamification is an emerging technique to enhance motivation and performance in traditionally unengaging tasks like software testing. Previous studies have indicated that gamified systems have the potential to improve software testing processes by providing testers with achievements and feedback. However, further evidence of these benefits across different environments, programming languages, and participant groups is required. This paper aims to replicate and validate the effects of IntelliGame, a gamification plugin for IntelliJ IDEA to engage developers in writing and executing tests. The objective is to generalize the benefits observed in earlier studies to new contexts, i.e., the TypeScript programming language and a larger participant pool. The replicability study consists of a controlled experiment with 174 participants, divided into two groups: one using IntelliGame and one with no gamification plugin. The study employed a two-group experimental design to compare testing behavior, coverage, mutation scores, and participant feedback between the groups. Data was collected through test metrics and participant surveys, and statistical analysis was performed to determine the statistical significance. Participants using IntelliGame showed higher engagement and productivity in testing practices than the control group, evidenced by the creation of more tests, increased frequency of executions, and enhanced utilization of testing tools. This ultimately led to better code implementations, highlighting the effectiveness of gamification in improving functional outcomes and motivating users in their testing endeavors. The replication study confirms that gamification, through IntelliGame, positively impacts software testing behavior and developer engagement in coding tasks.
Paper Structure (42 sections, 5 figures, 1 table)

This paper contains 42 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Gamification in Software Testing
  4. IntelliGame
  5. Replicability, Reproducibility, and Repeatability
  6. A Replicability Study of the Initial IntelliGame Study
  7. Research Questions
  8. Object Selection
  9. Pilot Study
  10. Experiment Design
  11. Participant Selection
  12. Experimental Analysis
  13. RQ1: Does IntelliGame influence testing behaviour?
  14. RQ2: How does IntelliGame influence resulting test suites?
  15. RQ3: Do achievement levels reflect differences in test suites and activities?
  16. ...and 27 more sections

Figures (5)

  • Figure 1: IntelliGame window showing part of the achievements and their progress
  • Figure 2: Differences between control and treatment groups over time
  • Figure 3: Differences between control and treatment groups
  • Figure 4: The number of achievement levels, their Pearson correlations with different test suite metrics as well as the ratios of failed tests and test smells
  • Figure 5: Survey responses of the participants as Likert plots