Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Non-Optimal Decisions of Embodied Agents via Diversity-Guided Metamorphic Testing

Wenzhao Wu, Yahui Tang, Mingfei Cheng, Wenbing Tang, Yuan Zhou, Yang Liu

TL;DR

This work identifies and formalizes the problem of Non-optimal Decisions (NoDs), where agents complete tasks successfully but inefficiently, and introduces a diversity-guided selection strategy that actively selects test cases exploring different violation categories, avoiding redundant evaluations while ensuring comprehensive diversity coverage.

Abstract

As embodied agents advance toward real-world deployment, ensuring optimal decisions becomes critical for resource-constrained applications. Current evaluation methods focus primarily on functional correctness, overlooking the non-functional optimality of generated plans. This gap can lead to significant performance degradation and resource waste. We identify and formalize the problem of Non-optimal Decisions (NoDs), where agents complete tasks successfully but inefficiently. We present NoD-DGMT, a systematic framework for detecting NoDs in embodied agent task planning via diversity-guided metamorphic testing. Our key insight is that optimal planners should exhibit invariant behavioral properties under specific transformations. We design four novel metamorphic relations capturing fundamental optimality properties: position detour suboptimality, action optimality completeness, condition refinement monotonicity, and scene perturbation invariance. To maximize detection efficiency, we introduce a diversity-guided selection strategy that actively selects test cases exploring different violation categories, avoiding redundant evaluations while ensuring comprehensive diversity coverage. Extensive experiments on the AI2-THOR simulator with four state-of-the-art planning models demonstrate that NoD-DGMT achieves violation detection rates of 31.9% on average, with our diversity-guided filter improving rates by 4.3% and diversity scores by 3.3 on average. NoD-DGMT significantly outperforms six baseline methods, with 16.8% relative improvement over the best baseline, and demonstrates consistent superiority across different model architectures and task complexities.

Detecting Non-Optimal Decisions of Embodied Agents via Diversity-Guided Metamorphic Testing

TL;DR

This work identifies and formalizes the problem of Non-optimal Decisions (NoDs), where agents complete tasks successfully but inefficiently, and introduces a diversity-guided selection strategy that actively selects test cases exploring different violation categories, avoiding redundant evaluations while ensuring comprehensive diversity coverage.

Abstract

As embodied agents advance toward real-world deployment, ensuring optimal decisions becomes critical for resource-constrained applications. Current evaluation methods focus primarily on functional correctness, overlooking the non-functional optimality of generated plans. This gap can lead to significant performance degradation and resource waste. We identify and formalize the problem of Non-optimal Decisions (NoDs), where agents complete tasks successfully but inefficiently. We present NoD-DGMT, a systematic framework for detecting NoDs in embodied agent task planning via diversity-guided metamorphic testing. Our key insight is that optimal planners should exhibit invariant behavioral properties under specific transformations. We design four novel metamorphic relations capturing fundamental optimality properties: position detour suboptimality, action optimality completeness, condition refinement monotonicity, and scene perturbation invariance. To maximize detection efficiency, we introduce a diversity-guided selection strategy that actively selects test cases exploring different violation categories, avoiding redundant evaluations while ensuring comprehensive diversity coverage. Extensive experiments on the AI2-THOR simulator with four state-of-the-art planning models demonstrate that NoD-DGMT achieves violation detection rates of 31.9% on average, with our diversity-guided filter improving rates by 4.3% and diversity scores by 3.3 on average. NoD-DGMT significantly outperforms six baseline methods, with 16.8% relative improvement over the best baseline, and demonstrates consistent superiority across different model architectures and task complexities.
Paper Structure (23 sections, 5 equations, 5 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 5 equations, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Embodied Task Planning
  4. Non-Optimal Decisions (NoDs)
  5. Metamorphic Testing (MT)
  6. Motivation and Problem Statement
  7. Motivation
  8. Problem Statement
  9. Methodology
  10. Overview
  11. Metamorphic Relations (MRs)
  12. Diversity-Guided Filter
  13. Metamorphic Testing-based Violation Detection Framework
  14. Evaluation
  15. Experimental Settings
  16. ...and 8 more sections

Figures (5)

  • Figure 1: The task planning process of embodied agents.
  • Figure 2: A motivating example revealing an optimality issue in embodied agent decision-making. The navigation target is marked with a red circle, and the agent's terminal position is indicated by a white circle.
  • Figure 3: The three-phase pipeline of NoD-DGMT: test case generation, diversity-guided selection, and violation detection.
  • Figure 4: Four types of mutations and their corresponding metamorphic relations for NoD detection. Gray, green, and red circles denote starting positions, intermediate waypoints, and target positions, respectively.
  • Figure 5: Typical failure cases of NoD detected by NoD-DGMT.

Theorems & Definitions (1)

  • Definition 1: NoDs