"Estimating software project effort using analogies": Reflections after 28 years

TL;DR

The paper revisits a seminal 1997 study that proposed analogy-based software effort estimation via case-based reasoning implemented in the ANGEL tool, using $k$ nearest donors with $2 \le k \le 5$ and a distance metric to predict effort. It reports out-of-sample validation across nine diverse datasets (275 projects) with LOOCV, showing that analogy-based prediction is competitive with or superior to a linear regression benchmark while avoiding information leakage. In a long-form reflection, the author analyzes what was achieved, what endured, and what could have been improved, emphasizing the importance of data sharing, explicit reporting of variance, and robust validation. The retrospective highlights enduring methodological principles—rigorous out-of-sample evaluation, data/tool openness, and explainability—while calling for Open Science practices, effect-size reporting, and comprehensive threats-to-validity assessments to guide future software engineering prediction research.

Abstract

Background: This invited paper is the result of an invitation to write a retrospective article on a "TSE most influential paper" as part of the journal's 50th anniversary. Objective: To reflect on the progress of software engineering prediction research using the lens of a selected, highly cited research paper and 28 years of hindsight. Methods: The paper examines (i) what was achieved, (ii) what has endured and (iii) what could have been done differently with the benefit of retrospection. Conclusions: While many specifics of software project effort prediction have evolved, key methodological issues remain relevant. The original study emphasised empirical validation with benchmarks, out-of-sample testing and data/tool sharing. Four areas for improvement are identified: (i) stronger commitment to Open Science principles, (ii) focus on effect sizes and confidence intervals, (iii) reporting variability alongside typical results and (iv) more rigorous examination of threats to validity.