SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separation

TL;DR

This work tackles the gap between distortion-based metrics and human perceptual judgments in open-domain audio separation by introducing SAJ, a reference-free, multimodal evaluation framework. SAJ supports text, visual, and span prompts to predict four perceptual dimensions—recall, precision, faithfulness, and overall quality—and is trained on large-scale human judgments across speech, music, and general sounds. The approach demonstrates strong alignment with human ratings, outperforms a range of baselines, and enables practical uses such as data filtering, pseudo-labeling, and reranking at scale. By integrating semantic, spatial, and temporal cues, SAJ provides a robust, scalable alternative to traditional metrics, with potential to guide future multimodal evaluation and model development in audio separation.

Abstract

The performance evaluation remains a complex challenge in audio separation, and existing evaluation metrics are often misaligned with human perception, course-grained, relying on ground truth signals. On the other hand, subjective listening tests remain the gold standard for real-world evaluation, but they are expensive, time-consuming, and difficult to scale. This paper addresses the growing need for automated systems capable of evaluating audio separation without human intervention. The proposed evaluation metric, SAM Audio Judge (SAJ), is a multimodal fine-grained reference-free objective metric, which shows highly alignment with human perceptions. SAJ supports three audio domains (speech, music and general sound events) and three prompt inputs (text, visual and span), covering four different dimensions of evaluation (recall, percision, faithfulness, and overall). SAM Audio Judge also shows potential applications in data filtering, pseudo-labeling large datasets and reranking in audio separation models. We release our code and pre-trained models at: https://github.com/facebookresearch/sam-audio.

Figures (6)

• Figure 1: Distribution of recall, precision, faithfulness, and overall scores across the speech, music, and sound modalities.
• Figure 2: Joint score distributions between recall, precision, faithfulness, and overall ratings. The horizontal axes are normalized for each metric, allowing direct comparison of their correlation patterns with the overall rating.
• Figure 3: Overview of the SAJ model. Given the input audio and output audio, SAJ predicts the separation performance in four dimensions (recall, precision, faithfulness and overall), conditioned on any combination of text descriptions (text prompts), visual masks (visual prompts), and temporal intervals (span prompts).
• Figure 4: Separation performance comparison between different rerankers. We use 8 candidates for the reranking in SAM Audio samaudio.
• Figure 5: Human overall score as a function of the SAJ filtering threshold. Each point corresponds to retaining the top-$m\%$ highest-rated samples under SAJ, with the percentage indicating the remaining data after filtering. The red dashed line denotes the pseudo-labeled data used in SAM Audio samaudio.