Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SourceBroken: A large-scale analysis on the (un)reliability of SourceRank in the PyPI ecosystem

Biagio Montaruli, Serena Elisa Ponta, Luca Compagna, Davide Balzarotti

TL;DR

SourceRank aggregates $18$ metrics to rate open-source packages, but this work shows significant reliability gaps under evasion attacks in PyPI. It presents a threat model and identifies URL confusion as a major vector that can affect $5$ of the metrics, enabling malicious packages to masquerade as trustworthy. Empirically, SourceRank poorly distinguishes benign from malicious packages in real-world data due to delayed removal signaling, though it performs well on MalwareBench when removals are reflected; URL confusion can drive high scores (up to $15$) when combined with other evasion tactics. The authors propose mitigations such as periodic removal status checks and robust URL verification, and they discuss extending the analysis to other ecosystems (e.g., npm) and to other scoring systems to improve defense of software supply chains.

Abstract

SourceRank is a scoring system made of 18 metrics that assess the popularity and quality of open-source packages. Despite being used in several recent studies, none has thoroughly analyzed its reliability against evasion attacks aimed at inflating the score of malicious packages, thereby masquerading them as trustworthy. To fill this gap, we first propose a threat model that identifies potential evasion approaches for each metric, including the URL confusion technique, which can affect 5 out of the 18 metrics by leveraging a URL pointing to a legitimate repository potentially unrelated to the malicious package. Furthermore, we study the reliability of SourceRank in the PyPI ecosystem by analyzing the SourceRank distributions of benign and malicious packages in the state-of-the-art MalwareBench dataset, as well as in a real-world dataset of 122,398 packages. Our analysis reveals that, while historical data suggests a clear distinction between benign and malicious packages, the real-world distributions overlap significantly, mainly due to SourceRank's failure to timely reflect package removals. As a result, SourceRank cannot be reliably used to discriminate between benign and malicious packages in real-world scenarios, nor to select benign packages among those available on PyPI. Finally, our analysis reveals that URL confusion represents an emerging attack vector, with its prevalence increasing from 4.2% in MalwareBench to 7.0% in our real-world dataset. Moreover, this technique is often used alongside other evasion techniques and can significantly inflate the SourceRank metrics of malicious packages.

SourceBroken: A large-scale analysis on the (un)reliability of SourceRank in the PyPI ecosystem

TL;DR

SourceRank aggregates metrics to rate open-source packages, but this work shows significant reliability gaps under evasion attacks in PyPI. It presents a threat model and identifies URL confusion as a major vector that can affect of the metrics, enabling malicious packages to masquerade as trustworthy. Empirically, SourceRank poorly distinguishes benign from malicious packages in real-world data due to delayed removal signaling, though it performs well on MalwareBench when removals are reflected; URL confusion can drive high scores (up to ) when combined with other evasion tactics. The authors propose mitigations such as periodic removal status checks and robust URL verification, and they discuss extending the analysis to other ecosystems (e.g., npm) and to other scoring systems to improve defense of software supply chains.

Abstract

SourceRank is a scoring system made of 18 metrics that assess the popularity and quality of open-source packages. Despite being used in several recent studies, none has thoroughly analyzed its reliability against evasion attacks aimed at inflating the score of malicious packages, thereby masquerading them as trustworthy. To fill this gap, we first propose a threat model that identifies potential evasion approaches for each metric, including the URL confusion technique, which can affect 5 out of the 18 metrics by leveraging a URL pointing to a legitimate repository potentially unrelated to the malicious package. Furthermore, we study the reliability of SourceRank in the PyPI ecosystem by analyzing the SourceRank distributions of benign and malicious packages in the state-of-the-art MalwareBench dataset, as well as in a real-world dataset of 122,398 packages. Our analysis reveals that, while historical data suggests a clear distinction between benign and malicious packages, the real-world distributions overlap significantly, mainly due to SourceRank's failure to timely reflect package removals. As a result, SourceRank cannot be reliably used to discriminate between benign and malicious packages in real-world scenarios, nor to select benign packages among those available on PyPI. Finally, our analysis reveals that URL confusion represents an emerging attack vector, with its prevalence increasing from 4.2% in MalwareBench to 7.0% in our real-world dataset. Moreover, this technique is often used alongside other evasion techniques and can significantly inflate the SourceRank metrics of malicious packages.
Paper Structure (10 sections, 4 figures, 3 tables)

This paper contains 10 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. SourceRank Threat Modeling
  3. SourceRank Overview
  4. Threat Model
  5. Experiments
  6. Research Questions
  7. Datasets
  8. Experimental Setup
  9. Results
  10. Conclusions and Future Work

Figures (4)

  • Figure 1: SourceRank score for the pandas package provided on the Libraries.io webpage (left) and obtained through the Libraries.io API as JSON (right).
  • Figure 2: SourceRank distributions of benign (blue) and malicious (red) packages for (a) MalwareBench and (b) real-world datasets. The other color in the plot (purple), is given by the overlapping between the blue and red distributions.
  • Figure 3: SourceRank distributions of benign and malicious packages for the real-world dataset after updating the Is Removed metric.
  • Figure 4: Tuning of the SourceRank threshold for detection of malicious packages on the target datasets: MalwareBench (blue) and real-world (red for the original SourceRank, yellow for the one obtained after updating the Is Removed metric).