Creating benchmarkable components to measure the quality ofAI-enhanced developer tools
Elise Paradis, Ambar Murillo, Maulishree Pandey, Sarah D'Angelo, Matthew Hughes, Andrew Macvean, Ben Ferrari-Church
TL;DR
The paper addresses the absence of benchmark standards for the developer experience (DX) in genAI-enabled coding tools. It presents an end-to-end process to create robust, enterprise-grade, modular DX benchmarking components, including sentiment and productivity metrics, attitude surveys, a benchmarkable coding task, and a detailed study protocol. The instruments are validated through literature review and two rounds of cognitive testing, culminating in a three-phase randomized experiment to measure AI impact on developer velocity. The authors report ROI estimates, cross-feature comparisons, and practical guidance for adoption, demonstrating how DX benchmarks can balance model quality with user experience and enable industry-wide comparability.
Abstract
In the AI community, benchmarks to evaluate model quality are well established, but an equivalent approach to benchmarking products built upon generative AI models is still missing. This has had two consequences. First, it has made teams focus on model quality over the developer experience, while successful products combine both. Second, product team have struggled to answer questions about their products in relation to their competitors. In this case study, we share: (1) our process to create robust, enterprise-grade and modular components to support the benchmarking of the developer experience (DX) dimensions of our team's AI for code offerings, and (2) the components we have created to do so, including demographics and attitudes towards AI surveys, a benchmarkable task, and task and feature surveys. By doing so, we hope to lower the barrier to the DX benchmarking of genAI-enhanced code products.