Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages

Daniel Venturini, Filipe Roseiro Cogo, Ivanilton Polato, Marco A Gerosa, Igor Scaliante Wiese

TL;DR

This study empirically examines manifesting breaking changes in the npm ecosystem, focusing on how non-major dependency updates can disrupt client builds and how clients recover. Using a mixed-method approach, the authors analyze metadata from over 1 million npm packages, test a randomized sample of 384 clients across thousands of releases, and manually verify failure cases to classify root causes and recovery paths. They find that roughly 12% of client packages and 14% of their releases are affected by manifesting breaking changes, with about 44% occurring in minor or patch releases. Recovery is common but varies by actor: providers fix most issues within a median of 7 days, transitive providers fix faster than direct clients, and upgrading the provider is the most frequent recovery method. The work provides actionable guidance for dependency management, testing, and documentation to reduce disruption and improve resilience in package ecosystems.

Abstract

Complex software systems have a network of dependencies. Developers often configure package managers (e.g., npm) to automatically update dependencies with each publication of new releases containing bug fixes and new features. When a dependency release introduces backward-incompatible changes, commonly known as breaking changes, dependent packages may not build anymore. This may indirectly impact downstream packages, but the impact of breaking changes and how dependent packages recover from these breaking changes remain unclear. To close this gap, we investigated the manifestation of breaking changes in the npm ecosystem, focusing on cases where packages' builds are impacted by breaking changes from their dependencies. We measured the extent to which breaking changes affect dependent packages. Our analyses show that around 12% of the dependent packages and 14% of their releases were impacted by a breaking change during updates of non-major releases of their dependencies. We observed that, from all of the manifesting breaking changes, 44% were introduced both in minor and patch releases, which in principle should be backward compatible. Clients recovered themselves from these breaking changes in half of the cases, most frequently by upgrading or downgrading the provider's version without changing the versioning configuration in the package manager. We expect that these results help developers understand the potential impact of such changes and recover from them.

I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages

TL;DR

This study empirically examines manifesting breaking changes in the npm ecosystem, focusing on how non-major dependency updates can disrupt client builds and how clients recover. Using a mixed-method approach, the authors analyze metadata from over 1 million npm packages, test a randomized sample of 384 clients across thousands of releases, and manually verify failure cases to classify root causes and recovery paths. They find that roughly 12% of client packages and 14% of their releases are affected by manifesting breaking changes, with about 44% occurring in minor or patch releases. Recovery is common but varies by actor: providers fix most issues within a median of 7 days, transitive providers fix faster than direct clients, and upgrading the provider is the most frequent recovery method. The work provides actionable guidance for dependency management, testing, and documentation to reduce disruption and improve resilience in package ecosystems.

Abstract

Complex software systems have a network of dependencies. Developers often configure package managers (e.g., npm) to automatically update dependencies with each publication of new releases containing bug fixes and new features. When a dependency release introduces backward-incompatible changes, commonly known as breaking changes, dependent packages may not build anymore. This may indirectly impact downstream packages, but the impact of breaking changes and how dependent packages recover from these breaking changes remain unclear. To close this gap, we investigated the manifestation of breaking changes in the npm ecosystem, focusing on cases where packages' builds are impacted by breaking changes from their dependencies. We measured the extent to which breaking changes affect dependent packages. Our analyses show that around 12% of the dependent packages and 14% of their releases were impacted by a breaking change during updates of non-major releases of their dependencies. We observed that, from all of the manifesting breaking changes, 44% were introduced both in minor and patch releases, which in principle should be backward compatible. Clients recovered themselves from these breaking changes in half of the cases, most frequently by upgrading or downgrading the provider's version without changing the versioning configuration in the package manager. We expect that these results help developers understand the potential impact of such changes and recover from them.
Paper Structure (26 sections, 5 figures, 9 tables)

This paper contains 26 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Definitions, Scope and Motivating Examples
  3. Glossary definitions
  4. Motivating Examples
  5. Study Design
  6. Data Collection
  7. Obtaining metadata from npm packages
  8. Running clients' tests
  9. Manual check on failure cases: detecting manifesting breaking changes
  10. Research questions: motivation, approach
  11. RQ1. To what extent do manifesting breaking changes manifest in client packages?
  12. RQ2. What problems in the provider package cause a manifesting breaking change?
  13. RQ3. How do client packages recover from a manifesting breaking change?
  14. Scope and Limitations
  15. Results
  16. ...and 11 more sections

Figures (5)

  • Figure 1: An example of a dependency tree, with clients and providers.
  • Figure 2: The evolution of the dependency tree (resolved versions) for ember-cli-chartjs@2.1.1 when it was published (middle-tree) and when the associated tests with the release were executed in our study (right-hand tree).
  • Figure 3: Approach to sampling the database and executing the associated tests with the client release.
  • Figure 4: Proportion of fixed/recovered manifesting breaking changes by provider and client packages and the respective Semantic Version level of the fixing/recovering releases.
  • Figure 5: Provider's version changed by clients and transitive providers. On the left side of each figure, one can see the range level where the manifesting breaking change was introduced and on the right side, one can see the range level where the same manifesting breaking change was fixed.