Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards a Fault-Injection Benchmarking Suite

Tianhao Wang, Robin Thunig, Horst Schirmeier

TL;DR

The paper addresses the lack of a dedicated benchmarking suite for fault-tolerance (FT) and fault-injection (FI) research, arguing that using benchmarks from other domains yields poor comparability and inefficiency. It proposes a set of preferable FT/FI benchmark properties, including multiple granularities, domain and program-characteristic classification, resource-efficient FI infrastructure, and a self-contained runtime to improve cross-study comparability. An illustrative evaluation with MiBench and TACLeBench demonstrates how runtime choice can drastically affect fault-space characteristics and failure modes, reinforcing the need for representative benchmarks and richer characterization. The work aims to foster a standard for FT/FI benchmarks that enables scalable, comparable, and efficient FI campaigns across the research community.

Abstract

Soft errors in memories and logic circuits are known to disturb program execution. In this context, the research community has been proposing a plethora of fault-tolerance (FT) solutions over the last decades, as well as fault-injection (FI) approaches to test, measure and compare them. However, there is no agreed-upon benchmarking suite for demonstrating FT or FI approaches. As a replacement, authors pick benchmarks from other domains, e.g. embedded systems. This leads to little comparability across publications, and causes behavioral overlap within benchmarks that were not selected for orthogonality in the FT/FI domain. In this paper, we want to initiate a discussion on what a benchmarking suite for the FT/FI domain should look like, and propose criteria for benchmark selection.

Towards a Fault-Injection Benchmarking Suite

TL;DR

The paper addresses the lack of a dedicated benchmarking suite for fault-tolerance (FT) and fault-injection (FI) research, arguing that using benchmarks from other domains yields poor comparability and inefficiency. It proposes a set of preferable FT/FI benchmark properties, including multiple granularities, domain and program-characteristic classification, resource-efficient FI infrastructure, and a self-contained runtime to improve cross-study comparability. An illustrative evaluation with MiBench and TACLeBench demonstrates how runtime choice can drastically affect fault-space characteristics and failure modes, reinforcing the need for representative benchmarks and richer characterization. The work aims to foster a standard for FT/FI benchmarks that enables scalable, comparable, and efficient FI campaigns across the research community.

Abstract

Soft errors in memories and logic circuits are known to disturb program execution. In this context, the research community has been proposing a plethora of fault-tolerance (FT) solutions over the last decades, as well as fault-injection (FI) approaches to test, measure and compare them. However, there is no agreed-upon benchmarking suite for demonstrating FT or FI approaches. As a replacement, authors pick benchmarks from other domains, e.g. embedded systems. This leads to little comparability across publications, and causes behavioral overlap within benchmarks that were not selected for orthogonality in the FT/FI domain. In this paper, we want to initiate a discussion on what a benchmarking suite for the FT/FI domain should look like, and propose criteria for benchmark selection.
Paper Structure (4 sections, 1 figure)

This paper contains 4 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Preferable Benchmark Properties
  3. Evaluation of Selected Properties
  4. Conclusion

Figures (1)

  • Figure 1: Number of dynamic instructions, unique memory-access locations and SDCs (circle sizes) of MiBenchguthaus:2001:mibench benchmarks compiled with picolibcrepo:picolibc:1.8.6 and TACLeBenchTACLeBench as used by Borchert et al. borchert:23:dsn, measured with the FAIL* fault-injection framework.