Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Invariance in Agentic AI

I. de Zarzà, J. de Curtò, Jordi Cabot, Pietro Manzoni, Carlos T. Calafate

Abstract

Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance. Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.

Semantic Invariance in Agentic AI

Abstract

Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance. Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.
Paper Structure (15 sections, 6 equations, 5 figures, 4 tables)

This paper contains 15 sections, 6 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Structural Transformations
  5. Verbosity Transformations
  6. Contextual Transformations
  7. Evaluation Metrics
  8. Solution-Level Metrics
  9. Step-Level Metrics
  10. Trace-Level Metrics
  11. Aggregate Robustness Metrics
  12. Experimental Setup
  13. Problem Corpus
  14. Results
  15. Conclusion

Figures (5)

  • Figure 1: Metamorphic relations organized by transformation category. Each card shows original problem text and its semantic-preserving transformation, with key modifications highlighted.
  • Figure 2: Metamorphic relation taxonomy and implementation.
  • Figure 3: Robustness analysis showing Mean Absolute Delta (lower = more robust), score change distributions, semantic similarity of reasoning steps, and score change ranges across models.
  • Figure 4: Heatmaps showing mean score delta (left) and semantic similarity (right) by metamorphic relation and model. Darker red indicates performance degradation; darker blue indicates higher semantic consistency.
  • Figure 5: Score delta distributions by metamorphic relation and model. Box plots show median, interquartile range, and outliers. The contrastive transformation induces the widest variance across all models, while identity and paraphrase show tightest distributions for robust models.

Theorems & Definitions (1)

  • definition 1: Semantic Invariance