Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unpacking Security Scanners for GitHub Actions Workflows

Madjda Fares, Yogya Gamage, Benoit Baudry

TL;DR

The paper conducts the first systematic comparison of nine static GitHub Actions workflow security scanners, introducing a 10-weakness taxonomy and benchmarking on 596 real-world workflows. It demonstrates that scanners vary widely in scope, rule granularity, and performance, with no single tool covering all weaknesses and with significant cross-tool reporting differences. The authors recommend a pragmatic combination of tools (e.g., actionlint+poutine as a baseline, plus autofix-focused scanners like frizbee) to maximize coverage while keeping CI costs low, and they provide open data for reproducibility. This work informs developers and security teams about tool selection and integration strategies to harden GitHub Actions workflows against software supply chain attacks.

Abstract

GitHub Actions is a widely used platform that allows developers to automate the build and deployment of their projects through configurable workflows. As the platform's popularity continues to grow, it has become a target of choice for recent software supply chain attacks. These attacks exploit excessive permissions, ambiguous versions, or the absence of artifact integrity checks to compromise workflows. In response to these attacks, several security scanners have emerged to help developers harden their workflows. In this paper, we perform the first systematic comparison of 9 GitHub Actions workflow security scanners. We compare them in terms of scope (which security weaknesses they target), detection capabilities (how many weaknesses they detect), and usability (how long they take to scan a workflow). To compare scanners on a common ground, we first establish a taxonomy of 10 security weaknesses that can occur in GitHub Actions workflows. Then, we run the scanners against a curated set of 596 workflows. Our study reveals that the landscape of GitHub Actions workflow security scanners is diverse, with both broad-scope tools and very focused ones. More importantly, we show that scanners interpret security weaknesses differently, leading to significant differences in the type and number of reported weaknesses. Based on this empirical evidence, we make actionable recommendations for developers to harden their GitHub Actions workflows.

Unpacking Security Scanners for GitHub Actions Workflows

TL;DR

The paper conducts the first systematic comparison of nine static GitHub Actions workflow security scanners, introducing a 10-weakness taxonomy and benchmarking on 596 real-world workflows. It demonstrates that scanners vary widely in scope, rule granularity, and performance, with no single tool covering all weaknesses and with significant cross-tool reporting differences. The authors recommend a pragmatic combination of tools (e.g., actionlint+poutine as a baseline, plus autofix-focused scanners like frizbee) to maximize coverage while keeping CI costs low, and they provide open data for reproducibility. This work informs developers and security teams about tool selection and integration strategies to harden GitHub Actions workflows against software supply chain attacks.

Abstract

GitHub Actions is a widely used platform that allows developers to automate the build and deployment of their projects through configurable workflows. As the platform's popularity continues to grow, it has become a target of choice for recent software supply chain attacks. These attacks exploit excessive permissions, ambiguous versions, or the absence of artifact integrity checks to compromise workflows. In response to these attacks, several security scanners have emerged to help developers harden their workflows. In this paper, we perform the first systematic comparison of 9 GitHub Actions workflow security scanners. We compare them in terms of scope (which security weaknesses they target), detection capabilities (how many weaknesses they detect), and usability (how long they take to scan a workflow). To compare scanners on a common ground, we first establish a taxonomy of 10 security weaknesses that can occur in GitHub Actions workflows. Then, we run the scanners against a curated set of 596 workflows. Our study reveals that the landscape of GitHub Actions workflow security scanners is diverse, with both broad-scope tools and very focused ones. More importantly, we show that scanners interpret security weaknesses differently, leading to significant differences in the type and number of reported weaknesses. Based on this empirical evidence, we make actionable recommendations for developers to harden their GitHub Actions workflows.
Paper Structure (20 sections, 3 figures, 5 tables)

This paper contains 20 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Terminology of GitHub Actions
  3. Workflow Security Scanners under Study
  4. Sourcing Open Source Workflow Security Scanners
  5. Curated list of Workflow Security Scanners
  6. Taxonomy of GitHub Actions Workflows Security weaknesses
  7. Methodology
  8. Taxonomy of security weaknesses
  9. Methodology
  10. Research questions
  11. Dataset of GitHub Actions Workflows
  12. Results
  13. To what extent do state of the art workflow security scanners search for the same workflow weaknesses?
  14. To what extent do state of the art workflow security scanners behave consistently regarding the detection of security weaknesses?
  15. What are the runtime performances and reporting characteristics of security scanners?
  16. ...and 5 more sections

Figures (3)

  • Figure 1: An example of a GitHub Actions Workflow. Excerpt from the https://github.com/excalidraw/excalidraw/blob/master/.github/workflows/size-limit.yml workflow from https://github.com/excalidraw/excalidraw/tree/master.
  • Figure 2: Examples to illustrate our taxonomy of 10 high-level security weaknesses in GitHub Actions Workflows
  • Figure 3: Execution time distribution across workflows for each tool (sorted by number of LOC).