Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Constraint-Guided Test Execution Scheduling: An Experience Report at ABB Robotics

Arnaud Gotlieb, Morten Mossige, Helge Spieker

TL;DR

The paper tackles automated test execution scheduling in CI for industrial robotic systems with hardware/software integration. It documents DynTest, a constraint-programming–based approach that evolves from greedy scheduling to a CP model using global constraints and rotational diversity. An empirical evaluation in ABB's IPS subsystem demonstrates high resource utilization and effective test prioritization, illustrating practical benefits and transferability. The work highlights the value of industry–academic collaboration in transferring advanced constraint-based optimization techniques to production environments.

Abstract

Automated test execution scheduling is crucial in modern software development environments, where components are frequently updated with changes that impact their integration with hardware systems. Building test schedules, which focus on the right tests and make optimal use of the available resources, both time and hardware, under consideration of vast requirements on the selection of test cases and their assignment to certain test execution machines, is a complex optimization task. Manual solutions are time-consuming and often error-prone. Furthermore, when software and hardware components and test scripts are frequently added, removed or updated, static test execution scheduling is no longer feasible and the motivation for automation taking care of dynamic changes grows. Since 2012, our work has focused on transferring technology based on constraint programming for automating the testing of industrial robotic systems at ABB Robotics. After having successfully transferred constraint satisfaction models dedicated to test case generation, we present the results of a project called DynTest whose goal is to automate the scheduling of test execution from a large test repository, on distinct industrial robots. This paper reports on our experience and lessons learned for successfully transferring constraint-based optimization models for test execution scheduling at ABB Robotics. Our experience underlines the benefits of a close collaboration between industry and academia for both parties.

Constraint-Guided Test Execution Scheduling: An Experience Report at ABB Robotics

TL;DR

The paper tackles automated test execution scheduling in CI for industrial robotic systems with hardware/software integration. It documents DynTest, a constraint-programming–based approach that evolves from greedy scheduling to a CP model using global constraints and rotational diversity. An empirical evaluation in ABB's IPS subsystem demonstrates high resource utilization and effective test prioritization, illustrating practical benefits and transferability. The work highlights the value of industry–academic collaboration in transferring advanced constraint-based optimization techniques to production environments.

Abstract

Automated test execution scheduling is crucial in modern software development environments, where components are frequently updated with changes that impact their integration with hardware systems. Building test schedules, which focus on the right tests and make optimal use of the available resources, both time and hardware, under consideration of vast requirements on the selection of test cases and their assignment to certain test execution machines, is a complex optimization task. Manual solutions are time-consuming and often error-prone. Furthermore, when software and hardware components and test scripts are frequently added, removed or updated, static test execution scheduling is no longer feasible and the motivation for automation taking care of dynamic changes grows. Since 2012, our work has focused on transferring technology based on constraint programming for automating the testing of industrial robotic systems at ABB Robotics. After having successfully transferred constraint satisfaction models dedicated to test case generation, we present the results of a project called DynTest whose goal is to automate the scheduling of test execution from a large test repository, on distinct industrial robots. This paper reports on our experience and lessons learned for successfully transferring constraint-based optimization models for test execution scheduling at ABB Robotics. Our experience underlines the benefits of a close collaboration between industry and academia for both parties.
Paper Structure (11 sections, 4 figures)

This paper contains 11 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Test Execution Scheduling at ABB Robotics
  3. Automated Testing Process
  4. Data Initialization and Acquisition
  5. Test Case Prioritization
  6. Selection and Scheduling
  7. Distribution and Execution
  8. Reporting
  9. Empirical Evaluation
  10. Lessons Learned
  11. Conclusion

Figures (4)

  • Figure 1: Overview of the CI cycle and the challenge related to time management
  • Figure 2: The test controller distributes individual test plans to each test agent, which controls, in turn, one robot and records log information and test outcomes.
  • Figure 3: Visualization of a test schedule with interactive access to test results.
  • Figure 4: Visualization of a test report for the IPS project and distribution of resource usage and test case priority among CI cycles for a one-month period.