Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models

Kai Li, Can Shen, Yile Liu, Jirui Han, Kelong Zheng, Xuechao Zou, Zhe Wang, Shun Zhang, Xingjian Du, Hanjun Luo, Yingbin Jin, Xinxin Xing, Ziyang Ma, Yue Liu, Yifan Zhang, Junfeng Fang, Kun Wang, Yibo Yan, Gelei Deng, Haoyang Li, Yiming Li, Xiaobin Zhuang, Tianlong Chen, Qingsong Wen, Tianwei Zhang, Yang Liu, Haibo Hu, Zhizheng Wu, Xiaolin Hu, Eng-Siong Chng, Wenyuan Xu, XiaoFeng Wang, Wei Dong, Xinfeng Li

TL;DR

AudioTrust presents the first end-to-end benchmark dedicated to evaluating the multifaceted trustworthiness of Audio Large Language Models (ALLMs). By introducing six audio-specific dimensions—Fairness, Hallucination, Safety, Privacy, Robustness, and Authentication—and 26 subtasks across 4,420 samples, it reveals systematic weaknesses in both open- and closed-source ALLMs under realistic, high-risk audio conditions. The framework combines a modular two-stage platform with GPT-4o-based automated evaluation and targeted human validation to deliver scalable, reproducible insights into how acoustic cues influence model behavior. Key contributions include a comprehensive dataset, a robust evaluation protocol, and public leaderboards that illuminate disparities across model families, guiding safer deployment and ongoing improvement of ALLMs in safety-critical contexts. The work highlights actionable avenues for debiasing, stronger privacy protections against paralinguistic leakage, improved robustness to audio distortions, and fortified defenses against voice-based attacks, establishing AudioTrust as a foundational benchmark for trustworthy audio AI systems.

Abstract

Audio Large Language Models (ALLMs) have gained widespread adoption, yet their trustworthiness remains underexplored. Existing evaluation frameworks, designed primarily for text, fail to address unique vulnerabilities introduced by audio's acoustic properties. We identify significant trustworthiness risks in ALLMs arising from non-semantic acoustic cues, including timbre, accent, and background noise, which can manipulate model behavior. We propose AudioTrust, a comprehensive framework for systematic evaluation of ALLM trustworthiness across audio-specific risks. AudioTrust encompasses six key dimensions: fairness, hallucination, safety, privacy, robustness, and authentication. The framework implements 26 distinct sub-tasks using a curated dataset of over 4,420 audio samples from real-world scenarios, including daily conversations, emergency calls, and voice assistant interactions. We conduct comprehensive evaluations across 18 experimental configurations using human-validated automated pipelines. Our evaluation of 14 state-of-the-art open-source and closed-source ALLMs reveals significant limitations when confronted with diverse high-risk audio scenarios, providing insights for secure deployment of audio models. Code and data are available at https://github.com/JusperLee/AudioTrust.

AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Models

TL;DR

AudioTrust presents the first end-to-end benchmark dedicated to evaluating the multifaceted trustworthiness of Audio Large Language Models (ALLMs). By introducing six audio-specific dimensions—Fairness, Hallucination, Safety, Privacy, Robustness, and Authentication—and 26 subtasks across 4,420 samples, it reveals systematic weaknesses in both open- and closed-source ALLMs under realistic, high-risk audio conditions. The framework combines a modular two-stage platform with GPT-4o-based automated evaluation and targeted human validation to deliver scalable, reproducible insights into how acoustic cues influence model behavior. Key contributions include a comprehensive dataset, a robust evaluation protocol, and public leaderboards that illuminate disparities across model families, guiding safer deployment and ongoing improvement of ALLMs in safety-critical contexts. The work highlights actionable avenues for debiasing, stronger privacy protections against paralinguistic leakage, improved robustness to audio distortions, and fortified defenses against voice-based attacks, establishing AudioTrust as a foundational benchmark for trustworthy audio AI systems.

Abstract

Audio Large Language Models (ALLMs) have gained widespread adoption, yet their trustworthiness remains underexplored. Existing evaluation frameworks, designed primarily for text, fail to address unique vulnerabilities introduced by audio's acoustic properties. We identify significant trustworthiness risks in ALLMs arising from non-semantic acoustic cues, including timbre, accent, and background noise, which can manipulate model behavior. We propose AudioTrust, a comprehensive framework for systematic evaluation of ALLM trustworthiness across audio-specific risks. AudioTrust encompasses six key dimensions: fairness, hallucination, safety, privacy, robustness, and authentication. The framework implements 26 distinct sub-tasks using a curated dataset of over 4,420 audio samples from real-world scenarios, including daily conversations, emergency calls, and voice assistant interactions. We conduct comprehensive evaluations across 18 experimental configurations using human-validated automated pipelines. Our evaluation of 14 state-of-the-art open-source and closed-source ALLMs reveals significant limitations when confronted with diverse high-risk audio scenarios, providing insights for secure deployment of audio models. Code and data are available at https://github.com/JusperLee/AudioTrust.

Paper Structure

This paper contains 76 sections, 12 equations, 10 figures, 27 tables.

Table of Contents

  1. Introduction
  2. AudioTrust: Fairness
  3. AudioTrust: Hallucination
  4. AudioTrust: Safety
  5. AudioTrust: Privacy
  6. AudioTrust: Robustness
  7. AudioTrust: Authentication
  8. Conclusions
  9. Dataset Curation and Human Participants
  10. Potential Misuse and Dual Use Concerns
  11. Introduction to Audio Large Language Models
  12. Speech Understanding Models
  13. Speech Interaction Models
  14. Benchmark Models
  15. Open-Source Models
  16. ...and 61 more sections

Figures (10)

  • Figure 1: AudioTrust's mission: evaluating and understanding multifaceted trustworthiness risks of audio large language models, and inspire secure and trustworthy deployment of future audio models.
  • Figure 2: (a) AudioTrust features 6 core trustworthiness dimensions, which are broken down into 26 specific sub-categories for granular evaluation. (b) Preliminary leaderboard showcasing the performance of 9 contemporary open- and closed-source ALLMs across these dimensions.
  • Figure 3: Overview of the unified trustworthiness evaluation framework for ALLMs, illustrating the decoupled two-stage architecture encompassing inference execution (Stage 1) and trustworthiness assessment (Stage 2).
  • Figure 4: A tree taxonomy of different perspectives of trustworthiness that AudioTrust focuses on.
  • Figure 5: Fairness experiment design concept
  • ...and 5 more figures