Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case

Joao Correia, Daniel Coutinho, Marco Castelluccio, Caio Barbosa, Rafael de Mello, Anita Sarma, Alessandro Garcia, Marco Gerosa, Igor Steinmacher

TL;DR

This study compares human, GPT-4o, and RAG-enhanced GPT-generated answers to 52 real Firefox developer questions sourced from Matrix chats, evaluated by eight Mozilla engineers on helpfulness, comprehensiveness, and conciseness. The results show RAG-based answers are more comprehensive than human answers while maintaining comparable helpfulness, and are often preferred in practice, though they tend to be verbose. GPT alone lags in comprehensiveness and sometimes in correctness due to lack of grounding. The findings suggest RAG grounding can reduce core maintainer burden in large OSS projects while delivering high-quality, context-aware assistance, with caveats about verbosity and generalizability beyond Firefox. Future work includes broader OSS evaluations, refined retrieval strategies, and integration with developer tools for improved usability and trust.

Abstract

The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.

A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case

TL;DR

This study compares human, GPT-4o, and RAG-enhanced GPT-generated answers to 52 real Firefox developer questions sourced from Matrix chats, evaluated by eight Mozilla engineers on helpfulness, comprehensiveness, and conciseness. The results show RAG-based answers are more comprehensive than human answers while maintaining comparable helpfulness, and are often preferred in practice, though they tend to be verbose. GPT alone lags in comprehensiveness and sometimes in correctness due to lack of grounding. The findings suggest RAG grounding can reduce core maintainer burden in large OSS projects while delivering high-quality, context-aware assistance, with caveats about verbosity and generalizability beyond Firefox. Future work includes broader OSS evaluations, refined retrieval strategies, and integration with developer tools for improved usability and trust.

Abstract

The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.
Paper Structure (33 sections, 3 figures, 5 tables)

This paper contains 33 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in Software Engineering
  4. Supporting Newcomers to Software Projects
  5. Improving LLM Performance with RAG
  6. Study Design
  7. Research Questions
  8. Data Collection
  9. Step 1: Data Collection
  10. Step 2: Question Filtering
  11. Expert Filtering.
  12. Step 3: Collecting Answers from Models
  13. Step 4: Expert Assessment of Answers
  14. Data Analysis
  15. Results and Discussions
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Study design (The numbers in parentheses refer to the volume of data in each stage of the method).
  • Figure 2: Number of answers and the occurrence of attributes
  • Figure 3: Correlation between answers attributes and their likelihood of being seen in practice.