OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
Lingkai Meng, Yu Shao, Long Yuan, Longbin Lai, Peng Cheng, Wenyuan Yu, Wenjie Zhang, Xuemin Lin, Jingren Zhou
TL;DR
Open source software usability evaluation is hampered by cost and scalability of human studies. The authors present OSS-UAgent, an automated, agent-based framework that leverages LLM-powered Researcher, Developer, Code Generator, and Evaluator to simulate developers of varying experience, build a platform-specific knowledge base, generate code, and assess usability with multi-dimensional metrics (Compliance, Correctness, Readability). A key novelty is anonymizing platform identifiers to ensure fair, generalizable evaluation and introducing a Compliance metric to capture adherence to platform standards. The demonstration on graph analytics platforms shows how the framework automates data collection, knowledge-base construction, code generation, and reporting, offering a scalable tool for OSS usability assessment with potential applicability to other domains.
Abstract
Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation.