Framework Matters: Energy Efficiency of UI Automation Testing Frameworks

TL;DR

This study quantifies per-action energy consumption across four web UI automation frameworks (Nightwatch, Playwright, Puppeteer, Selenium) in a controlled Raspberry Pi setup. It finds that energy costs vary by both UI action and framework, with up to about a sixfold difference, and that a significant interaction between framework and action type shapes energy use. Puppeteer generally yields the lowest energy footprint, while Nightwatch tends to be least efficient; no framework is best for all actions. The results underscore the need for per-framework calibration and per-action energy reporting to enable energy-aware test design and fair cross-framework comparisons, with broad implications for sustainable web UI testing and tooling choices.

Abstract

We examine per action energy consumption across four web user interface (UI) automation testing frameworks to determine whether consistent tendencies can guide energy-aware test design. Using a controlled client-server setup with external power metering, we repeat each UI action (refresh, click variants, checkbox, drag&drop, input-text, scroll) 35 times. Across each of the actions, energy costs vary by both framework and action. Puppeteer is the most efficient for left-click, right-click, double-click, checkbox, and input-text; Selenium is the most efficient for refresh and scroll; Nightwatch is generally the least energy efficient. The energy cost of performing the same action varied by up to a factor of six depending on the framework. This indicates that providing transparency of energy consumption for UI automation testing frameworks allows developers to make informed, energy-aware decisions when testing a specific UI action.

Figures (6)

• Figure 1: Visual verification of distribution shapes for energy consumption across framework action combinations. The plots reveal that many of the framework action combinations deviate from normality.
• Figure 2: Visual summary of Shapiro-Wilk and Anderson-Darling tests. Green indicates a pass in both cases. Red indicates a failure in both cases. Yellow indicates a pass for one test and fail in the other. Gray indicates a non-posible action.
• Figure 3: Estimated energy cost per UI action grouped by the different frameworks, where The x-axis lists the different UI actions and the y-axis shows the energy consumption per interaction in joules (J).
• Figure 4: Pairwise significance of energy consumption differences between UI actions based on Dunn’s test with Bonferroni correction. Higher values indicate stronger statistical significance after correction. Darker cells highlight pairs with the largest differences.
• Figure 5: Estimated energy cost per UI action, with the different frameworks represented by different colors. The x-axis lists the different UI automation frameworks and the y-axis shows the energy consumption per interaction in Joules.