Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human and Machine: How Software Engineers Perceive and Engage with AI-Assisted Code Reviews Compared to Their Peers

Adam Alami, Neil A. Ernst

TL;DR

This paper investigates software engineers' perceptions and engagement with AI-assisted code reviews versus human peer reviews. It uses a qualitative interview study with 20 engineers, employing both peer feedback and ChatGPT-generated reviews to probe cognitive, emotional, and behavioral responses. The findings show that while LLM-assisted feedback reduces emotional strain due to its consistent, professional tone, it can impose higher cognitive load due to dense, detailed content; trust and codebase context constrain adoption of AI feedback. The study highlights the importance of emotional intelligence, constructive feedback, and personalized, hybrid human-AI collaboration as the path forward for integrating AI into SE practices with social and practical benefits.

Abstract

The integration of artificial intelligence (AI) continues to increase and evolve, including in software engineering (SE). This integration involves processes traditionally entrusted to humans, such as coding. However, the impact on socio-technical processes like code review remains underexplored. In this interview-based study (20 interviewees), we investigate how software engineers perceive and engage with Large Language Model (LLM)-assisted code reviews compared to human peer-led reviews. In this inherently human-centric process, we aim to understand how software engineers navigate the introduction of AI into collaborative workflows. We found that engagement in code review is multi-dimensional, spanning cognitive, emotional, and behavioral dimensions. The introduction of LLM-assisted review impacts some of these attributes. For example, there is less need for emotional regulation and coping mechanisms when dealing with an LLM compared to peers. However, the cognitive load sometimes is higher in dealing with LLM-generated feedback due to its excessive details. Software engineers use a similar sense-making process to evaluate and adopt feedback suggestions from their peers and the LLM. However, the LLM feedback adoption is constrained by trust and lack of context in the review. Our findings contribute to a deeper understanding of how AI tools are impacting SE socio-technical processes and provide insights into the future of AI-human collaboration in SE practices.

Human and Machine: How Software Engineers Perceive and Engage with AI-Assisted Code Reviews Compared to Their Peers

TL;DR

This paper investigates software engineers' perceptions and engagement with AI-assisted code reviews versus human peer reviews. It uses a qualitative interview study with 20 engineers, employing both peer feedback and ChatGPT-generated reviews to probe cognitive, emotional, and behavioral responses. The findings show that while LLM-assisted feedback reduces emotional strain due to its consistent, professional tone, it can impose higher cognitive load due to dense, detailed content; trust and codebase context constrain adoption of AI feedback. The study highlights the importance of emotional intelligence, constructive feedback, and personalized, hybrid human-AI collaboration as the path forward for integrating AI into SE practices with social and practical benefits.

Abstract

The integration of artificial intelligence (AI) continues to increase and evolve, including in software engineering (SE). This integration involves processes traditionally entrusted to humans, such as coding. However, the impact on socio-technical processes like code review remains underexplored. In this interview-based study (20 interviewees), we investigate how software engineers perceive and engage with Large Language Model (LLM)-assisted code reviews compared to human peer-led reviews. In this inherently human-centric process, we aim to understand how software engineers navigate the introduction of AI into collaborative workflows. We found that engagement in code review is multi-dimensional, spanning cognitive, emotional, and behavioral dimensions. The introduction of LLM-assisted review impacts some of these attributes. For example, there is less need for emotional regulation and coping mechanisms when dealing with an LLM compared to peers. However, the cognitive load sometimes is higher in dealing with LLM-generated feedback due to its excessive details. Software engineers use a similar sense-making process to evaluate and adopt feedback suggestions from their peers and the LLM. However, the LLM feedback adoption is constrained by trust and lack of context in the review. Our findings contribute to a deeper understanding of how AI tools are impacting SE socio-technical processes and provide insights into the future of AI-human collaboration in SE practices.
Paper Structure (20 sections, 1 figure, 3 tables)

This paper contains 20 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Interviewee recruitment & selection
  5. Data collection
  6. Data analysis
  7. Ethics and informed consent
  8. Shared documents package
  9. Findings
  10. Feedback
  11. Engagement
  12. Reviewer context
  13. Human-AI collaboration
  14. Discussion
  15. Emotional intelligence
  16. ...and 5 more sections

Figures (1)

  • Figure 1: An abstract presentation of the findings, capturing engagement and how engineers respond to feedback. The line from Reviewer Context to Peer and LLM indicates two type of reviewers, and the ovals inside the rectangles are elements of the reviewer contexts.