Providing Information About Implemented Algorithms Improves Program Comprehension: A Controlled Experiment

TL;DR

This study empirically evaluates whether annotating source code with algorithm labels (name plus reference) improves program comprehension. In a controlled experiment with 56 participants across four Java-based exercises in the CoCoME codebase, the labeled condition yielded a significant improvement in correctness (median gain of $6$ points, about $23\%$, $p=0.040$) but did not reduce completion time ($p=0.991$). Qualitative data show that participants view labels as helpful for understanding intent and identify use cases such as error detection, optimization, and library replacement. The findings suggest algorithm labels can meaningfully enhance comprehension, particularly for developers with medium experience, and point to practical directions for integrating recognition cues into development tools and workflows.

Abstract

Context: Various approaches aim to support program comprehension by automatically detecting algorithms in source code. However, no empirical evaluations of their helpfulness have been performed. Objective: To empirically evaluate how algorithm labels - which include the algorithm's name and additional information - impact program comprehension in terms of correctness and time. Method: We conducted a controlled experiment with 56 participants, where the experimental group received code with labeled algorithms. The groups completed exercises designed to measure program comprehension as well as a post-questionnaire on label helpfulness, use cases for algorithm recognition, and reasons for self-implementation of algorithms in practice. Results: Annotating source code with algorithm labels significantly improves program comprehension (p=0.040), with a median improvement of 6 points (~23%), but does not affect completion times (p=0.991). Qualitative analysis revealed that a majority of participants perceived the labels as helpful, especially for recognizing the codes intent. Participants also proposed use cases such as error detection, optimization, and library replacement. Reasons for self-implementing algorithms included library inadequacies, performance needs and avoiding dependencies or licensing costs. Conclusion: This study shows that algorithm labels improve program comprehension, especially for developers with medium programming experience. Our qualitative analysis also sheds light on how participants benefit from the labels, further use cases for algorithm recognition and motivations behind self-implementing algorithms.

Figures (9)

• Figure 1: Overview of the study procedure.
• Figure 2: Program comprehension scores by group.
• Figure 3: Interaction plot showing the effect of algorithm labels on the mean program comprehension scores across different levels of prior programming experience.
• Figure 4: Completion time across all exercises by group.
• Figure 5: Distribution of the completion times for exercise one (leftmost) to exercise four (rightmost). Dashed lines indicate the time limit reminders. Note that the x-axis is scaled differently because some exercises were shorter than others.