Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Before Autonomy Takes Control: Software Testing in Robotics

Nils Chur, Thiago Santos de Moura, Argentina Ortega, Sven Peldszus, Thorsten Berger, Nico Hochgeschwender, Yannic Noller

TL;DR

Robotic systems pose safety-critical testing challenges due to tight hardware–software coupling and non-deterministic environments. The authors perform a systematic mapping of robotics testing papers to established software testing theory, guided by ISO 29119, to reveal how robotics testing aligns with or diverges from conventional practice. They find a heavy emphasis on system-level, simulation- and field-based testing, with limited unit/integration testing and sparse white-box techniques, highlighting a gap between robotics practice and software engineering standards. The study advocates standards-driven, structured testing processes, earlier-stage testing, and hybrid simulation–real-world strategies to improve test coverage, reproducibility, and trust in robotic systems.

Abstract

Robotic systems are complex and safety-critical software systems. As such, they need to be tested thoroughly. Unfortunately, robot software is intrinsically hard to test compared to traditional software, mainly since the software needs to closely interact with hardware, account for uncertainty in its operational environment, handle disturbances, and act highly autonomously. However, given the large space in which robots operate, anticipating possible failures when designing tests is challenging. This paper presents a mapping study by considering robotics testing papers and relating them to the software testing theory. We consider 247 robotics testing papers and map them to software testing, discussing the state-of-the-art software testing in robotics with an illustrated example, and discuss current challenges. Forming the basis to introduce both the robotics and software engineering communities to software testing challenges. Finally, we identify open questions and lessons learned.

Before Autonomy Takes Control: Software Testing in Robotics

TL;DR

Robotic systems pose safety-critical testing challenges due to tight hardware–software coupling and non-deterministic environments. The authors perform a systematic mapping of robotics testing papers to established software testing theory, guided by ISO 29119, to reveal how robotics testing aligns with or diverges from conventional practice. They find a heavy emphasis on system-level, simulation- and field-based testing, with limited unit/integration testing and sparse white-box techniques, highlighting a gap between robotics practice and software engineering standards. The study advocates standards-driven, structured testing processes, earlier-stage testing, and hybrid simulation–real-world strategies to improve test coverage, reproducibility, and trust in robotic systems.

Abstract

Robotic systems are complex and safety-critical software systems. As such, they need to be tested thoroughly. Unfortunately, robot software is intrinsically hard to test compared to traditional software, mainly since the software needs to closely interact with hardware, account for uncertainty in its operational environment, handle disturbances, and act highly autonomously. However, given the large space in which robots operate, anticipating possible failures when designing tests is challenging. This paper presents a mapping study by considering robotics testing papers and relating them to the software testing theory. We consider 247 robotics testing papers and map them to software testing, discussing the state-of-the-art software testing in robotics with an illustrated example, and discuss current challenges. Forming the basis to introduce both the robotics and software engineering communities to software testing challenges. Finally, we identify open questions and lessons learned.
Paper Structure (22 sections, 6 figures, 1 table)

This paper contains 22 sections, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Testing
  4. A Case Study: Restaurant Robots
  5. Methodology
  6. Test Environment, Levels, Types, and Activities
  7. Test Environment
  8. Test Levels
  9. Test Types
  10. Functional Testing
  11. Non-Functional Testing
  12. Test Activities
  13. Test Design Techniques
  14. Black-Box Testing
  15. White-Box Testing
  16. ...and 7 more sections

Figures (6)

  • Figure 1: PAL Robotics TIAGo robot macenski2023regulated
  • Figure 2: A self-adaptive decentralized robotic architecture (SERA) garcia2018icsa
  • Figure 3: Overview of our search strategy and selection process
  • Figure 4: Number of robotics papers reporting about the respective testing aspect
  • Figure 5: Test levels in the V-model and examples of related parts of a robotic system
  • ...and 1 more figures