Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoMT: A Multi-Agent LLM Framework for Automated Metamorphic Testing of Autonomous Driving Systems

Linfeng Liang, Chenkai Tan, Yao Deng, Yingfeng Cai, T. Y Chen, Xi Zheng

TL;DR

AutoMT addresses the oracle problem in autonomous driving testing by automating metamorphic testing through a multi-agent LLM framework. It automatically extracts metamorphic relations from local traffic rules, grounds them in a RAG-based MR repository, analyzes real-world test cases with a vision-language agent, and generates valid follow-up scenarios via image editing and video synthesis. The approach yields greater MR diversity, higher validation rates, and improved detection of behavioral safety violations compared to expert-defined baselines, highlighting its practical impact for industrial ADS testing. The modular, ontology-driven pipeline is readily integrable into existing pipelines and can be extended to simulation-based testing to systematically cover underrepresented or safety-critical scenarios.

Abstract

Autonomous Driving Systems (ADS) are safety-critical, where failures can be severe. While Metamorphic Testing (MT) is effective for fault detection in ADS, existing methods rely heavily on manual effort and lack automation. We present AutoMT, a multi-agent MT framework powered by Large Language Models (LLMs) that automates the extraction of Metamorphic Relations (MRs) from local traffic rules and the generation of valid follow-up test cases. AutoMT leverages LLMs to extract MRs from traffic rules in Gherkin syntax using a predefined ontology. A vision-language agent analyzes scenarios, and a search agent retrieves suitable MRs from a RAG-based database to generate follow-up cases via computer vision. Experiments show that AutoMT achieves up to 5 x higher test diversity in follow-up case generation compared to the best baseline (manual expert-defined MRs) in terms of validation rate, and detects up to 20.55% more behavioral violations. While manual MT relies on a fixed set of predefined rules, AutoMT automatically extracts diverse metamorphic relations that augment real-world datasets and help uncover corner cases often missed during in-field testing and data collection. Its modular architecture separating MR extraction, filtering, and test generation supports integration into industrial pipelines and potentially enables simulation-based testing to systematically cover underrepresented or safety-critical scenarios.

AutoMT: A Multi-Agent LLM Framework for Automated Metamorphic Testing of Autonomous Driving Systems

TL;DR

AutoMT addresses the oracle problem in autonomous driving testing by automating metamorphic testing through a multi-agent LLM framework. It automatically extracts metamorphic relations from local traffic rules, grounds them in a RAG-based MR repository, analyzes real-world test cases with a vision-language agent, and generates valid follow-up scenarios via image editing and video synthesis. The approach yields greater MR diversity, higher validation rates, and improved detection of behavioral safety violations compared to expert-defined baselines, highlighting its practical impact for industrial ADS testing. The modular, ontology-driven pipeline is readily integrable into existing pipelines and can be extended to simulation-based testing to systematically cover underrepresented or safety-critical scenarios.

Abstract

Autonomous Driving Systems (ADS) are safety-critical, where failures can be severe. While Metamorphic Testing (MT) is effective for fault detection in ADS, existing methods rely heavily on manual effort and lack automation. We present AutoMT, a multi-agent MT framework powered by Large Language Models (LLMs) that automates the extraction of Metamorphic Relations (MRs) from local traffic rules and the generation of valid follow-up test cases. AutoMT leverages LLMs to extract MRs from traffic rules in Gherkin syntax using a predefined ontology. A vision-language agent analyzes scenarios, and a search agent retrieves suitable MRs from a RAG-based database to generate follow-up cases via computer vision. Experiments show that AutoMT achieves up to 5 x higher test diversity in follow-up case generation compared to the best baseline (manual expert-defined MRs) in terms of validation rate, and detects up to 20.55% more behavioral violations. While manual MT relies on a fixed set of predefined rules, AutoMT automatically extracts diverse metamorphic relations that augment real-world datasets and help uncover corner cases often missed during in-field testing and data collection. Its modular architecture separating MR extraction, filtering, and test generation supports integration into industrial pipelines and potentially enables simulation-based testing to systematically cover underrepresented or safety-critical scenarios.
Paper Structure (32 sections, 1 equation, 8 figures, 9 tables)

This paper contains 32 sections, 1 equation, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ADS Metamorphic Testing
  4. Automated MR Generation
  5. APPROACH
  6. Overview of AutoMT
  7. Metamorphic Relation Extraction Agent (M-Agent)
  8. LLM-based rule parser
  9. MR Validation
  10. Metamorphic Relation RAG Database
  11. Test Case Analysis Agent (T-Agent)
  12. Follow-up Test Case Generation Agent (F-Agent)
  13. MR Match
  14. Image Editing
  15. EXPERIMENT
  16. ...and 17 more sections

Figures (8)

  • Figure 1: A high-level overview of AutoMT.
  • Figure 2: The workflow of M-Agent
  • Figure 3: The manipulation operation distribution in the valid follow-up test case generated by AutoMT (blue) and MT with Manual MR (red).
  • Figure 4: Comparison of the original test case, follow-up test case generated by AutoMT and baselines.
  • Figure 5: Comparison of the original test case and follow-up test cases generated by MT with manual MR and AutoMT.
  • ...and 3 more figures