Classification of descriptions and summary using multiple passes of statistical and natural language toolkits

TL;DR

The paper tackles the problem of judging whether a Python package name is relevant to its PyPI summary. It proposes a multi-pass approach that starts with a baseline membership test and progressively adds dynamic n-grams, lemmatization, and fuzzy matching to address abbreviations, misspellings, and partial words. The results show substantial gains: zero-score entries drop from 362 (baseline) to 50 (3rd pass), while high relevance scores improve, with manual validation indicating the 3rd pass offers the best alignment with human judgments. The work demonstrates that name-relevance scoring can complement other metrics for final classification and suggests preprocessing and modeling enhancements for further improvements.

Abstract

This document describes a possible approach that can be used to check the relevance of a summary / definition of an entity with respect to its name. This classifier focuses on the relevancy of an entity's name to its summary / definition, in other words, it is a name relevance check. The percentage score obtained from this approach can be used either on its own or used to supplement scores obtained from other metrics to arrive upon a final classification; at the end of the document, potential improvements have also been outlined. The dataset that this document focuses on achieving an objective score is a list of package names and their respective summaries (sourced from pypi.org).

Figures (9)

• Figure 1: Incremental changes across consecutive attempts.
• Figure 2: Baseline attempt pipeline.
• Figure 3: Scores from Baseline Attempt
• Figure 4: 2nd attempt pipeline.
• Figure 5: Scores from 2nd Attempt