Automatic Categorization of GitHub Actions with Transformers and Few-shot Learning

TL;DR

The paper tackles the problem of making GitHub Actions more discoverable by automatically categorizing actions into semantic categories. It introduces Gavel, a transformer-based pipeline with few-shot learning that uses README.MD content to assign potentially multiple categories. An empirical evaluation on a merged dataset of $1{,}213$ actions across $30$ categories shows Gavel outperforms a CNBN baseline in $P$, $R$, and $F1$, though it requires more computation. The work provides a replication package and suggests future directions toward lightweight models and per-type tokenizers to further enhance practicality and efficiency.

Abstract

In the GitHub ecosystem, workflows are used as an effective means to automate development tasks and to set up a Continuous Integration and Delivery (CI/CD pipeline). GitHub Actions (GHA) have been conceived to provide developers with a practical tool to create and maintain workflows, avoiding reinventing the wheel and cluttering the workflow with shell commands. Properly leveraging the power of GitHub Actions can facilitate the development processes, enhance collaboration, and significantly impact project outcomes. To expose actions to search engines, GitHub allows developers to assign them to one or more categories manually. These are used as an effective means to group actions sharing similar functionality. Nevertheless, while providing a practical way to execute workflows, many actions have unclear purposes, and sometimes they are not categorized. In this work, we bridge such a gap by conceptualizing Gavel, a practical solution to increasing the visibility of actions in GitHub. By leveraging the content of README.MD files for each action, we use Transformer--a deep learning algorithm--to assign suitable categories to the action. We conducted an empirical investigation and compared Gavel with a state-of-the-art baseline. The experimental results show that our proposed approach can assign categories to GitHub actions effectively, thus outperforming the state-of-the-art baseline.

Figures (4)

• Figure 1: Examples of GitHub actions with and without categories.
• Figure 2: System architecture.
• Figure 3: One vs. all multi-label classification
• Figure 4: Comparison of accuracy between CNBN and Gavel.