Mind the Gap: The Missing Features of the Tools to Support User Studies in Software Engineering
Lázaro Costa, Susana Barbosa, Jácome Cunha
TL;DR
This paper investigates the gap between barriers faced in conducting user studies in software engineering and the features offered by existing tools. By mapping 18 Myers2023 barriers to 15 identified tool features and validating proposed new features with 102 researchers, the authors show that no current tool covers all barriers, though many features receive strong support. The study combines a comprehensive feature-barrier analysis with an online validation to produce actionable requirements for tool development, highlighting critical gaps in IRB-related support, data collection, and end-to-end orchestration. The work advances practical guidance for tool designers and provides a pathway toward more reproducible and scalable user studies in software engineering, with plans to implement a new tool and extend analyses to reproducibility packages.
Abstract
User studies are paramount for advancing science. However, researchers face several barriers when performing them despite the existence of supporting tools. In this work, we study how existing tools and their features cope with previously identified barriers. Moreover, we propose new features for the barriers that lack support. We validated our proposal with 102 researchers, achieving statistically significant positive support for all but one feature. We study the current gap between tools and barriers, using features as the bridge. We show there is a significant lack of support for several barriers, as some have no single tool to support them.