A Longitudinal Study of Usability in Identity-Based Software Signing

Abstract

Identity-based software signing tools aim to make software artifact provenance verifiable while reducing the operational burden of long-lived key management. However, there is limited cross-tool longitudinal evidence about which usability problems arise in practice and how those problems evolve as tools mature. This gap matters because unusable signing and verification workflows can lead to incomplete adoption, misconfiguration, or skipped verification, undermining intended integrity guarantees. We conducted the first mining-software-repositories study of five open-source identity-based signing ecosystems: Sigstore, OpenPubKey, HashiCorp Vault, Keyfactor, and Notary v2. We analyzed approximately 3,900 GitHub issues from Nov. 2021 to Nov. 2025. We coded each issue for the reported usability concern and the implicated architectural component, and compared patterns across tools and over time. Across ecosystems, reported concerns concentrate in verification workflows, policy and configuration surfaces, and integration boundaries. Longitudinal Poisson trend analysis shows substantial declines in reported issues for most ecosystems. However, across usability themes, workflow- and documentation-related concerns decline unevenly across tools and concern types, and verification workflows and configuration surfaces remain persistent friction points. These results indicate that identity-based signing reduces some usability burdens while relocating complexity to verification semantics, policy configuration, and deployment integration. Designing future signing ecosystems therefore requires treating verification semantics and release workflows as first-class usability targets rather than peripheral integration concerns.

Figures (7)

• Figure 1: Identity-based signing uses short-lived certificates instead of long-lived keys. Typical architectures decompose signing and verification across four components (\ref{['tab:id_components']}).
• Figure 2: Overview of our empirical pipeline for analyzing usability problems in identity-based software signing tools using GitHub issues. We show the stages of data collection and filtering, coding and LLM-assisted scaling, and downstream analysis used to answer RQ1 & RQ2 (problem categories and affected components) and RQ3 (changes over time).
• Figure 3: Heatmap of Poisson time-trend slopes ($\beta_1$) by tool and inductively generated primary usability theme. Negative values indicate decreasing expected monthly issue counts over time; positive values indicate increasing counts. Only statistically significant cells ($p<0.05$) are colorized and annotated with the estimated $\beta_1$; non-significant cells are masked.
• Figure 4: Aggregate Poisson regression curves of expected monthly issue counts across all tools over calendar time. Curves represent fitted Poisson means. Downward trajectories indicate decreasing expected counts (negative time slope), while upward trajectories indicate increasing expected counts (positive time slope).
• Figure 5: Traditional key-managed signing centers on a long-lived key pair that the signer creates and protects and that verifiers must obtain to validate signatures. The workflow highlights two recurring usability burdens: public-key distribution (often via a key server) and establishing confidence that the retrieved public key corresponds to the intended issuer identity. Compare this flow to that of identity-based signing, depicted in \ref{['fig:bg_id_workflow']}.