The Machine Vision Iceberg Explained: Advancing Dynamic Testing by Considering Holistic Environmental Relations

TL;DR

The paper addresses robustness gaps in machine-vision testing for highly automated driving by showing that existing evaluation strategies miss crucial environmental relations within the operating design domain. It introduces a holistic black-box testing framework that leverages seven MV-specific deficits, a Granularity Orders taxonomy to stratify environment-related information, and an Environmental Entity Relation Graph to visualize and reason about inter-object relations across scenarios. Through application examples focused on two deficits (D1 and D2) and analyses of pre-trained models, the work demonstrates how environmental relations can reveal failures not captured by standard scenario-based testing or benchmark datasets. The proposed approach aims to improve precision, efficiency, and completeness of MV testing and supports domain-shift detection and robust integration in HAD systems, with automation of environmental entity extraction identified as a key future step.

Abstract

Machine Vision (MV) is essential for solving driving automation. This paper examines potential shortcomings in current MV testing strategies for highly automated driving (HAD) systems. We argue for a more comprehensive understanding of the performance factors that must be considered during the MV evaluation process, noting that neglecting these factors can lead to significant risks. This is not only relevant to MV component testing, but also to integration testing. To illustrate this point, we draw an analogy to a ship navigating towards an iceberg to show potential hidden challenges in current MV testing strategies. The main contribution is a novel framework for black-box testing which observes environmental relations. This means it is designed to enhance MV assessments by considering the attributes and surroundings of relevant individual objects. The framework provides the identification of seven general concerns about the object recognition of MV, which are not addressed adequately in established test processes. To detect these deficits based on their performance factors, we propose the use of a taxonomy called "granularity orders" along with a graphical representation. This allows an identification of MV uncertainties across a range of driving scenarios. This approach aims to advance the precision, efficiency, and completeness of testing procedures for MV.

Figures (5)

• Figure 1: The strategies challenged by overcoming the metaphorical machine vision iceberg. While each can see the iceberg's hurdles from a high range of view, from a straight up view or a deep focus, none are able to see the full extent of the iceberg.
• Figure 2: General structure of the black-box testing framework incorporating selected solution from all three MV strategies by extending the analysis on performance factors relevant for MV with focus on environmental relations
• Figure 3: Taxonomy of granularity orders structuring explored performance factors of ODD's into seven orders based on their informational depth
• Figure 4: Schematic of the Environmental Entity Relation Graph
• Figure 5: Exemplary Application of the Environmental Entity Relation Graph