A Laboratory Experiment on Using Different Financial-Incentivization Schemes in Software-Engineering Experimentation

TL;DR

The paper investigates whether financial incentives influence developer performance in software-engineering experiments by combining a practitioner survey with a controlled lab experiment to compare payoff schemes that are performance-dependent, performance-independent, and open-source–like. Using a between-subject design and a code-review bug-detection task, the study finds observable behavioral differences across incentive schemes, but the primary metric, the $F1$-score, does not show statistically significant differences due to small sample sizes; time-on-task and engagement analyses offer additional insights. Eye-tracking is used as an exploratory validity check, indicating no adverse effects from wearing trackers. Overall, the work provides methodological guidance for incorporating incentives in SE experiments and highlights challenges in modeling open-source motivations, especially regarding external validity and metric selection. The authors also publish their artifacts to support replication and future work.

Abstract

In software-engineering research, many empirical studies are conducted with open-source or industry developers. However, in contrast to other research communities like economics or psychology, only few experiments use financial incentives (i.e., paying money) as a strategy to motivate participants' behavior and reward their performance. The most recent version of the SIGSOFT Empirical Standards mentions payouts only for increasing participation in surveys, but not for mimicking real-world motivations and behavior in experiments. Within this article, we report a controlled experiment in which we tackled this gap by studying how different financial incentivization schemes impact developers. For this purpose, we first conducted a survey on financial incentives used in the real-world, based on which we designed three incentivization schemes: (1) a performance-dependent scheme that employees prefer, (2) a scheme that is performance-independent, and (3) a scheme that mimics open-source development. Then, using a between-subject experimental design, we explored how these three schemes impact participants' performance. Our findings indicate that the different schemes can impact participants' performance in software-engineering experiments. Due to the small sample sizes, our results are not statistically significant, but we can still observe clear tendencies. Our contributions help understand the impact of financial incentives on participants in experiments as well as real-world scenarios, guiding researchers in designing experiments and organizations in compensating developers.

Figures (5)

• Figure 1: Screenshot of the code example as it will be shown we showed it to the participants. The checkboxes in front of each line allowed the participants to check buggy lines of code. Note that we will did not show the comments indicating the implemented bugs (i.e., in lines 16, 21, and 38). The blue boxes (not displayed to participants) indicate the Areas of Interest (AOIs) that we used for the eye-tracking analysis.
• Figure 2: Relation between sample size and Cohen's d for comparing two groups via the Wilcoxon- Mann-Whitney test, assuming a normal distribution with $\alpha = 0.0083$ and statistical power of 0.9.
• Figure 3: Boxplots for TP, TN, FP, and FN across our treatments. Each box shows the 25 % and 75 % quantiles as well as the median. The whiskers show the minimum and maximum values inside $1.5* IQR$. Outliers are displayed as points outside of the whiskers.
• Figure 4: Distribution of the completion times. The boxes show the 25 % and 75 % quantiles as well as the median. The whiskers show the minimum and maximum values inside $1.5* IQR$.
• Figure 5: Self-reported values of engagement, distress, and worry. The boxes show the 25 % and 75 % quantiles as well as the median. The whiskers show the minimum and maximum values inside $1.5* IQR$. Outliers are displayed as points outside of the whiskers.