Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards debiasing code review support

Tobias Jetzen, Xavier Devroey, Nicolas Matton, Benoît Vanderose

TL;DR

The paper addresses cognitive biases, specifically confirmation bias and decision fatigue, in code review and proposes a developer-centered design approach to debiasing. It develops iterative HTML prototypes, conducts two usability tests, and evaluates with the User Experience Questionnaire to identify effective techniques like Advice, pre-structured Form, and Guide, culminating in a final prototype that combines multiple techniques. The study offers a first design-oriented exploration of debiasing code review, reports qualitative and preliminary quantitative usability findings, and provides replication materials for broader validation. Its significance lies in outlining practical, tool-integrated interventions that could improve the objectivity and thoroughness of code reviews and informing future, broader evaluations across more biases and real-world settings.

Abstract

Cognitive biases appear during code review. They significantly impact the creation of feedback and how it is interpreted by developers. These biases can lead to illogical reasoning and decision-making, violating one of the main hypotheses supporting code review: developers' accurate and objective code evaluation. This paper explores harmful cases caused by cognitive biases during code review and potential solutions to avoid such cases or mitigate their effects. In particular, we design several prototypes covering confirmation bias and decision fatigue. We rely on a developer-centered design approach by conducting usability tests and validating the prototype with a user experience questionnaire (UEQ) and participants' feedback. We show that some techniques could be implemented in existing code review tools as they are well accepted by reviewers and help prevent behavior detrimental to code review. This work provides a solid first approach to treating cognitive bias in code review.

Towards debiasing code review support

TL;DR

The paper addresses cognitive biases, specifically confirmation bias and decision fatigue, in code review and proposes a developer-centered design approach to debiasing. It develops iterative HTML prototypes, conducts two usability tests, and evaluates with the User Experience Questionnaire to identify effective techniques like Advice, pre-structured Form, and Guide, culminating in a final prototype that combines multiple techniques. The study offers a first design-oriented exploration of debiasing code review, reports qualitative and preliminary quantitative usability findings, and provides replication materials for broader validation. Its significance lies in outlining practical, tool-integrated interventions that could improve the objectivity and thoroughness of code reviews and informing future, broader evaluations across more biases and real-world settings.

Abstract

Cognitive biases appear during code review. They significantly impact the creation of feedback and how it is interpreted by developers. These biases can lead to illogical reasoning and decision-making, violating one of the main hypotheses supporting code review: developers' accurate and objective code evaluation. This paper explores harmful cases caused by cognitive biases during code review and potential solutions to avoid such cases or mitigate their effects. In particular, we design several prototypes covering confirmation bias and decision fatigue. We rely on a developer-centered design approach by conducting usability tests and validating the prototype with a user experience questionnaire (UEQ) and participants' feedback. We show that some techniques could be implemented in existing code review tools as they are well accepted by reviewers and help prevent behavior detrimental to code review. This work provides a solid first approach to treating cognitive bias in code review.
Paper Structure (29 sections, 4 figures, 3 tables)

This paper contains 29 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Confirmation bias
  4. Decision fatigue
  5. Debiasing code review
  6. Confirmation bias
  7. Decision fatigue
  8. Design of the code review support
  9. First usability test
  10. Technique 1.1 –- Advice
  11. Technique 1.2 –- Form
  12. Technique 1.3 -– Guide
  13. Second usability test
  14. Technique 2.1 –- Advice
  15. Technique 2.2 –- Example
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Relationship between triggers, cognitive biases, and their effects.
  • Figure 2: Design, prototyping, and evaluation
  • Figure 3: Combinations of techniques.
  • Figure 4: Results of the UEQ benchmark for the advice with example (T3.1), the quick search with expert feedback (T3.2), and the guide (T3.3). Dots denote mean values, and error bars indicate 95% confidence intervals.