Agile Effort Estimation: Comparing the Accuracy and Efficiency of Planning Poker, Bucket System, and Affinity Estimation methods

TL;DR

The paper addresses the need to evaluate both accuracy and time efficiency in agile effort estimation by comparing Planning Poker, Bucket System, and Affinity Estimation within a capstone course using eight student teams and a fixed backlog. It employs Balanced Relative Error ($BRE$) to measure accuracy and observed time to measure efficiency, applying non-parametric tests and effect sizes. Results show Affinity Estimation is less accurate than Planning Poker and Bucket System, while Planning Poker and Bucket System have similar accuracy; Planning Poker is notably more time-consuming than the other methods. The Bucket System emerges as the best overall trade-off, offering comparable accuracy to Planning Poker with greater efficiency, though the study’s ESWS context warrants replication in industry settings to confirm external validity.

Abstract

Published studies on agile effort estimation predominantly focus on comparisons of the accuracy of different estimation methods, while efficiency comparisons, i.e. how much time the estimation methods consume was not in the forefront. However, for practical use in software development, the time required can be a very important cost factor for enterprises, especially when the accuracy of different agile effort estimations is similar. In this study, we thus try to advance the current standard accuracy comparison between methods by introducing efficiency i.e. time it takes to use a method as an additional dimension of comparison. We conduct this comparison between three agile effort estimation methods that were not yet compared in the literature, namely Planning Poker, Bucket System and Affinity Estimation. For the comparison, we used eight student teams with 29 students that had to use all the effort estimation methods during the course where they had to finish a programming project in 3 weeks. The results indicate that after the students get used to using the different methods the accuracy between them is not statistically significantly different, however, the efficiency is. On average Bucket System and Affinity Estimation methods take half as much time as Planning Poker.