Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Readability and Understandability of Snippets Recommended by General-purpose Web Search Engines: a Comparative Study

Carlos Eduardo C. Dantas, Marcelo A. Maia

TL;DR

This study addresses how readable and understandable code snippets are when retrieved from general-purpose web search engines. It collects top-5 results from Google, Bing, and Yahoo! for 9,480 queries, samples five sites, and evaluates code snippets using Scalabrino2018 readability and a normalized cognitive-complexity-based understandability metric, with results analyzed via ANOVA and Tukey post-hoc tests. The findings show that ranking has a small but significant effect on readability and understandability, with Google generally yielding more readable snippets and GeeksforGeeks providing the highest readability while Tutorialspoint offers the best understandability; site choice emerges as the strongest determinant of readability. These insights help developers and search-engine designers understand variability in code snippet quality across engines and sites, and point to avenues for broader site coverage and improved snippet extraction strategies.

Abstract

Developers often search for reusable code snippets on general-purpose web search engines like Google, Yahoo! or Microsoft Bing. But some of these code snippets may have poor quality in terms of readability or understandability. In this paper, we propose an empirical analysis to analyze the readability and understandability score from snippets extracted from the web using three independent variables: ranking, general-purpose web search engine, and recommended site. We collected the top-5 recommended sites and their respective code snippet recommendations using Google, Yahoo!, and Bing for 9,480 queries, and evaluate their readability and understandability scores. We found that some recommended sites have significantly better readability and understandability scores than others. The better-ranked code snippet is not necessarily more readable or understandable than a lower-ranked code snippet for all general-purpose web search engines. Moreover, considering the readability score, Google has better-ranked code snippets compared to Yahoo! or Microsoft Bing

Readability and Understandability of Snippets Recommended by General-purpose Web Search Engines: a Comparative Study

TL;DR

This study addresses how readable and understandable code snippets are when retrieved from general-purpose web search engines. It collects top-5 results from Google, Bing, and Yahoo! for 9,480 queries, samples five sites, and evaluates code snippets using Scalabrino2018 readability and a normalized cognitive-complexity-based understandability metric, with results analyzed via ANOVA and Tukey post-hoc tests. The findings show that ranking has a small but significant effect on readability and understandability, with Google generally yielding more readable snippets and GeeksforGeeks providing the highest readability while Tutorialspoint offers the best understandability; site choice emerges as the strongest determinant of readability. These insights help developers and search-engine designers understand variability in code snippet quality across engines and sites, and point to avenues for broader site coverage and improved snippet extraction strategies.

Abstract

Developers often search for reusable code snippets on general-purpose web search engines like Google, Yahoo! or Microsoft Bing. But some of these code snippets may have poor quality in terms of readability or understandability. In this paper, we propose an empirical analysis to analyze the readability and understandability score from snippets extracted from the web using three independent variables: ranking, general-purpose web search engine, and recommended site. We collected the top-5 recommended sites and their respective code snippet recommendations using Google, Yahoo!, and Bing for 9,480 queries, and evaluate their readability and understandability scores. We found that some recommended sites have significantly better readability and understandability scores than others. The better-ranked code snippet is not necessarily more readable or understandable than a lower-ranked code snippet for all general-purpose web search engines. Moreover, considering the readability score, Google has better-ranked code snippets compared to Yahoo! or Microsoft Bing

Paper Structure

This paper contains 13 sections, 1 equation, 5 figures.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Related Work
  4. Study Design
  5. Select Input Queries
  6. Collect Top-n Web Pages
  7. Extract Code Snippets
  8. Calculate Metrics
  9. Analysis Methods
  10. Results and Findings
  11. Discussion
  12. Threats to Validity
  13. Conclusions and Future Work

Figures (5)

  • Figure 1: Most popular sites with recommended web pages ranked on top-5 for 9,480 queries performed on Google, Microsoft Bing and Yahoo!
  • Figure 2: Tukey Test confidence intervals (x-axis) with the differences between ranking groups (y-axis)
  • Figure 3: Tukey Test confidence intervals (x-axis) with the differences between web search engines groups (y-axis)
  • Figure 4: Tukey Test confidence intervals (x-axis) with the differences between recommended sites groups (y-axis)
  • Figure 5: Distribution of understandability score