Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Energy Consumption of Test Generation

Fitsum Kifetew, Davide Prandi, Angelo Susi

TL;DR

The paper investigates the energy footprint of automated test generation using EvoSuite for Java programs, addressing how energy use differs across generation algorithms and class complexity, and comparing energy of generated versus manually written tests. It employs JoularJX energy measurement paired with RAPL-based profiling to capture per-method and per-test energy consumption under a fixed search budget. The findings show significant energy variation across algorithms, with higher energy when cyclomatic complexity is high and incomplete coverage, and that manual tests can be more energy-intensive without improving coverage. The work highlights concrete high-energy methods and suggests avenues for energy-aware test generation and broader applicability to other languages.

Abstract

Research in the area of automated test generation has seen remarkable progress in recent years, resulting in several approaches and tools for effective and efficient generation of test cases. In particular, the EvoSuite tool has been at the forefront of this progress embodying various algorithms for automated test generation of Java programs. EvoSuite has been used to generate test cases for a wide variety of programs as well. While there are a number of empirical studies that report results on the effectiveness, in terms of code coverage and other related metrics, of the various test generation strategies and algorithms implemented in EvoSuite, there are no studies, to the best of our knowledge, on the energy consumption associated to the automated test generation. In this paper, we set out to investigate this aspect by measuring the energy consumed by EvoSuite when generating tests. We also measure the energy consumed in the execution of the test cases generated, comparing them with those manually written by developers. The results show that the different test generation algorithms consumed different amounts of energy, in particular on classes with high cyclomatic complexity. Furthermore, we also observe that manual tests tend to consume more energy as compared to automatically generated tests, without necessarily achieving higher code coverage. Our results also give insight into the methods that consume significantly higher levels of energy, indicating potential points of improvement both for EvoSuite as well as the different programs under test.

On the Energy Consumption of Test Generation

TL;DR

The paper investigates the energy footprint of automated test generation using EvoSuite for Java programs, addressing how energy use differs across generation algorithms and class complexity, and comparing energy of generated versus manually written tests. It employs JoularJX energy measurement paired with RAPL-based profiling to capture per-method and per-test energy consumption under a fixed search budget. The findings show significant energy variation across algorithms, with higher energy when cyclomatic complexity is high and incomplete coverage, and that manual tests can be more energy-intensive without improving coverage. The work highlights concrete high-energy methods and suggests avenues for energy-aware test generation and broader applicability to other languages.

Abstract

Research in the area of automated test generation has seen remarkable progress in recent years, resulting in several approaches and tools for effective and efficient generation of test cases. In particular, the EvoSuite tool has been at the forefront of this progress embodying various algorithms for automated test generation of Java programs. EvoSuite has been used to generate test cases for a wide variety of programs as well. While there are a number of empirical studies that report results on the effectiveness, in terms of code coverage and other related metrics, of the various test generation strategies and algorithms implemented in EvoSuite, there are no studies, to the best of our knowledge, on the energy consumption associated to the automated test generation. In this paper, we set out to investigate this aspect by measuring the energy consumed by EvoSuite when generating tests. We also measure the energy consumed in the execution of the test cases generated, comparing them with those manually written by developers. The results show that the different test generation algorithms consumed different amounts of energy, in particular on classes with high cyclomatic complexity. Furthermore, we also observe that manual tests tend to consume more energy as compared to automatically generated tests, without necessarily achieving higher code coverage. Our results also give insight into the methods that consume significantly higher levels of energy, indicating potential points of improvement both for EvoSuite as well as the different programs under test.
Paper Structure (15 sections, 5 figures, 4 tables)

This paper contains 15 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Energy Profile of Sofware
  4. Automatic Test Generation
  5. Experiment Design
  6. Benchmark
  7. Empirical Procedure
  8. Performance metrics
  9. Results
  10. RQ1: Generation Cost
  11. RQ1.1: Methods Profile
  12. RQ2: Tests Execution Cost
  13. Discussion
  14. Threats To Validity
  15. Conclusions and Future Works

Figures (5)

  • Figure 1: Energy consumption stratified by full coverage achievement. The y axis represents the number (#) of classes. Note that data is not stratified per generation algorithm, and therefore classes are counted five times, one for each algorithm considered.
  • Figure 2: Boxplot of energy consumption per algorithm stratified by CCN and full coverage achievement.
  • Figure 3: Barplot of methods energy consumption per algorithm stratified by CCN and full coverage achievement.
  • Figure 4: Boxplot of the energy consumption of whole project test case generation (above) and execution (below). The y axis reports the $log_{10}$ of the Joule consumed. Projects names: prj_1 = 15058126273-mysql-table-helper, prj_2 = alipay-sofa-common-tools, prj_3 = amazonwebservices-aws-elasticache-cluster-client-memcached-for-java, prj_4 = carlosthe19916-sunat-web-services, prj_5 = CatDou-common-export, prj_6 = chrlembeck-chrlembeck-util, prj_7 = codelibs-jhighlight, prj_8 = easyinnova-Tiff-Library-4J
  • Figure 5: Scatter plot comparing the energy consumption of executing manually written (x-axis) and automatically generated (y-axis) tests for 162 projects. Each dot represents a project and its median energy consumption measured with JoularJX over 30 replicas. Selected outliers (blue points) are annotated with the branch coverage reached by manual (MC) and automatic (AC) tests. Note that the x and y axis are given in the $\log_{10}$ scale.