Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Models

Yuwen Wang, Xinyuan Qian, Tian-Hao Zhang, Jiaran Gao, Yuchen Pan, Xin Wang, Zhou Pan, Chen Wei, Yiming Wang

TL;DR

This work addresses the lack of personalization in large audio-language models by introducing PALM and the PALM-Bench benchmark. It formalizes PALM as conditional text generation P(Y|A,Q,P) and decomposes personalization into three tasks: concept activation, selective understanding, and personalized reasoning, evaluated via audio QA across single- and multi-speaker scenarios. PALM-Bench provides a large, bilingual dataset with rigorous single- and multi-speaker pipelines, enabling controlled evaluation with both training-free prompting and supervised fine-tuning strategies. The experiments reveal that existing prompting and fine-tuning methods yield limited personalized gains, though HD/AD cues and multi-speaker training can improve certain aspects, highlighting the need for memory, cross-task transfer, and robust personalization for real-world use.

Abstract

Large Audio-Language Models (LALMs) have demonstrated strong performance in audio understanding and generation. Yet, our extensive benchmarking reveals that their behavior is largely generic (e.g., summarizing spoken content) and fails to adequately support personalized question answering (e.g., summarizing what my best friend says). In contrast, human conditions their interpretation and decision-making on each individual's personal context. To bridge this gap, we formalize the task of Personalized LALMs (PALM) for recognizing personal concepts and reasoning within personal context. Moreover, we create the first benchmark (PALM-Bench) to foster the methodological advances in PALM and enable structured evaluation on several tasks across multi-speaker scenarios. Our extensive experiments on representative open-source LALMs, show that existing training-free prompting and supervised fine-tuning strategies, while yield improvements, remains limited in modeling personalized knowledge and transferring them across tasks robustly. Data and code will be released.

PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Models

TL;DR

This work addresses the lack of personalization in large audio-language models by introducing PALM and the PALM-Bench benchmark. It formalizes PALM as conditional text generation P(Y|A,Q,P) and decomposes personalization into three tasks: concept activation, selective understanding, and personalized reasoning, evaluated via audio QA across single- and multi-speaker scenarios. PALM-Bench provides a large, bilingual dataset with rigorous single- and multi-speaker pipelines, enabling controlled evaluation with both training-free prompting and supervised fine-tuning strategies. The experiments reveal that existing prompting and fine-tuning methods yield limited personalized gains, though HD/AD cues and multi-speaker training can improve certain aspects, highlighting the need for memory, cross-task transfer, and robust personalization for real-world use.

Abstract

Large Audio-Language Models (LALMs) have demonstrated strong performance in audio understanding and generation. Yet, our extensive benchmarking reveals that their behavior is largely generic (e.g., summarizing spoken content) and fails to adequately support personalized question answering (e.g., summarizing what my best friend says). In contrast, human conditions their interpretation and decision-making on each individual's personal context. To bridge this gap, we formalize the task of Personalized LALMs (PALM) for recognizing personal concepts and reasoning within personal context. Moreover, we create the first benchmark (PALM-Bench) to foster the methodological advances in PALM and enable structured evaluation on several tasks across multi-speaker scenarios. Our extensive experiments on representative open-source LALMs, show that existing training-free prompting and supervised fine-tuning strategies, while yield improvements, remains limited in modeling personalized knowledge and transferring them across tasks robustly. Data and code will be released.
Paper Structure (29 sections, 2 equations, 3 figures, 26 tables)

This paper contains 29 sections, 2 equations, 3 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Formulating LALMs Personalization
  4. Concept Activation (Recognition)
  5. Selective Understanding (Captioning)
  6. Personalized Reasoning
  7. The PALM-Bench Dataset
  8. Single-speaker Pipeline
  9. Multi-speaker Pipeline
  10. Statistics of PALM-Bench Dataset
  11. Evaluation on PALM-Bench
  12. Training-free Prompting Strategies
  13. Training-based Adaptation Strategies
  14. Conclusion
  15. Experimental Setup
  16. ...and 14 more sections

Figures (3)

  • Figure 1: General v.s. Personalized LALMs on personal concept recognition and personalized conversation.
  • Figure 2: Dataset creation and statistics. Left: The pipeline constructs diverse QA pairs grounded in speaker identities, covering (a) single- and (b) multi-speaker scenarios with negative sampling. (c) Detailed statistics showing the distribution of samples across speaker numbers, languages and durations.
  • Figure 3: Examples of different prompting strategies. Top panels (grey) show the distinct human and acoustic descriptions. Left (blue) and right (green) regions illustrate captioning and recognition tasks. Within the responses, bold black highlights reasoning based on the provided descriptions. Bold blue and bold green mark correct answers for their respective tasks, contrasting with bold red failures.