Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation

Xueyan Li, Yuxin Wang, Mengjie Jiang, Qingzi Zhu, Jiang Zhang, Zoey Kim, Yazhe Niu

TL;DR

The paper tackles the challenge of evaluating singing voice synthesis by introducing a generative reward framework that emits multi-dimensional language and audio critiques. It leverages a unified audio-language model trained on a hybrid of human reaction data and MLLM-generated feedback to produce interpretable assessments that guide model improvement. Through a joint text-audio training objective and careful data processing (including speech separation), the approach achieves robust evaluation signals and demonstrated expressive, human-like feedback. The work includes a practical benchmark and ablation studies, underscoring the value of combining diverse supervision sources for SVS evaluation.

Abstract

Singing voice synthesis (SVS) has advanced significantly, enabling models to generate vocals with accurate pitch and consistent style. As these capabilities improve, the need for reliable evaluation and optimization becomes increasingly critical. However, current methods like reward systems often rely on single numerical scores, struggle to capture various dimensions such as phrasing or expressiveness, and require costly annotations, limiting interpretability and generalization. To address these issues, we propose a generative feedback (i.e., reward model) framework that provides multi-dimensional language and audio feedback for SVS assessment. Our approach leverages an audio-language model to generate text and audio critiques-covering aspects such as melody, content, and auditory quality. The model is fine-tuned on a hybrid dataset combining human music reactions and synthetic critiques from a MLLMs, enhancing diversity and linguistic richness. Quantitative experiments validate the effectiveness of the proposed dataset and training strategy, demonstrating that the framework produces musically accurate and interpretable evaluations suitable for guiding generative model improvement. The code is at [https://github.com/opendilab/VocalCritic](https://github.com/opendilab/VocalCritic)

Generative Multi-modal Feedback for Singing Voice Synthesis Evaluation

TL;DR

The paper tackles the challenge of evaluating singing voice synthesis by introducing a generative reward framework that emits multi-dimensional language and audio critiques. It leverages a unified audio-language model trained on a hybrid of human reaction data and MLLM-generated feedback to produce interpretable assessments that guide model improvement. Through a joint text-audio training objective and careful data processing (including speech separation), the approach achieves robust evaluation signals and demonstrated expressive, human-like feedback. The work includes a practical benchmark and ablation studies, underscoring the value of combining diverse supervision sources for SVS evaluation.

Abstract

Singing voice synthesis (SVS) has advanced significantly, enabling models to generate vocals with accurate pitch and consistent style. As these capabilities improve, the need for reliable evaluation and optimization becomes increasingly critical. However, current methods like reward systems often rely on single numerical scores, struggle to capture various dimensions such as phrasing or expressiveness, and require costly annotations, limiting interpretability and generalization. To address these issues, we propose a generative feedback (i.e., reward model) framework that provides multi-dimensional language and audio feedback for SVS assessment. Our approach leverages an audio-language model to generate text and audio critiques-covering aspects such as melody, content, and auditory quality. The model is fine-tuned on a hybrid dataset combining human music reactions and synthetic critiques from a MLLMs, enhancing diversity and linguistic richness. Quantitative experiments validate the effectiveness of the proposed dataset and training strategy, demonstrating that the framework produces musically accurate and interpretable evaluations suitable for guiding generative model improvement. The code is at [https://github.com/opendilab/VocalCritic](https://github.com/opendilab/VocalCritic)

Paper Structure

This paper contains 21 sections, 2 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Dataset Construction
  4. Category 1: Human Reaction Data:
  5. Category 2: MLLM-generated Data:
  6. Model Training
  7. Experiments
  8. Main results:
  9. Multi-modal Supervision:
  10. Music Separation Ablation Study:
  11. Comparison between Human Reaction Data and MLLM-generated Data
  12. Dataset Details
  13. LLM-based Reaction Evaluation
  14. Experiment Settings
  15. Audio Case Study
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Overview of the data processing pipeline for constructing dual-source datasets.
  • Figure 2: The overview of the multi-modal reward model training and inferencing pipeline.
  • Figure 3: Comparison between Human Reaction Data and MLLM-generated Data.
  • Figure 4: Train loss and validation loss.
  • Figure 5: Loss curves for hybrid data training with Original vs. Separated Human data.