Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment

Huan Zhang, Vincent Cheung, Hayato Nishioka, Simon Dixon, Shinichi Furuya

TL;DR

This work introduces LLaQo, a Large Language Query-based music coach that leverages audio language modeling to provide detailed and formative assessments of music performances, and introduces instruction-tuned query-response datasets that cover a variety of performance dimensions from pitch accuracy to articulation.

Abstract

Research in music understanding has extensively explored composition-level attributes such as key, genre, and instrumentation through advanced representations, leading to cross-modal applications using large language models. However, aspects of musical performance such as stylistic expression and technique remain underexplored, along with the potential of using large language models to enhance educational outcomes with customized feedback. To bridge this gap, we introduce LLaQo, a Large Language Query-based music coach that leverages audio language modeling to provide detailed and formative assessments of music performances. We also introduce instruction-tuned query-response datasets that cover a variety of performance dimensions from pitch accuracy to articulation, as well as contextual performance understanding (such as difficulty and performance techniques). Utilizing AudioMAE encoder and Vicuna-7b LLM backend, our model achieved state-of-the-art (SOTA) results in predicting teachers' performance ratings, as well as in identifying piece difficulty and playing techniques. Textual responses from LLaQo was moreover rated significantly higher compared to other baseline models in a user study using audio-text matching. Our proposed model can thus provide informative answers to open-ended questions related to musical performance from audio data.

LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment

TL;DR

This work introduces LLaQo, a Large Language Query-based music coach that leverages audio language modeling to provide detailed and formative assessments of music performances, and introduces instruction-tuned query-response datasets that cover a variety of performance dimensions from pitch accuracy to articulation.

Abstract

Research in music understanding has extensively explored composition-level attributes such as key, genre, and instrumentation through advanced representations, leading to cross-modal applications using large language models. However, aspects of musical performance such as stylistic expression and technique remain underexplored, along with the potential of using large language models to enhance educational outcomes with customized feedback. To bridge this gap, we introduce LLaQo, a Large Language Query-based music coach that leverages audio language modeling to provide detailed and formative assessments of music performances. We also introduce instruction-tuned query-response datasets that cover a variety of performance dimensions from pitch accuracy to articulation, as well as contextual performance understanding (such as difficulty and performance techniques). Utilizing AudioMAE encoder and Vicuna-7b LLM backend, our model achieved state-of-the-art (SOTA) results in predicting teachers' performance ratings, as well as in identifying piece difficulty and playing techniques. Textual responses from LLaQo was moreover rated significantly higher compared to other baseline models in a user study using audio-text matching. Our proposed model can thus provide informative answers to open-ended questions related to musical performance from audio data.
Paper Structure (11 sections, 2 equations, 2 figures, 2 tables)

This paper contains 11 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Instruction tuning dataset on music education
  4. Model architecture and training
  5. Learning Objective
  6. Experiment setup
  7. Evaluation and discussions
  8. Compared baselines
  9. Objective evaluation
  10. Subjective feedback evaluation
  11. Conclusion and Future Work

Figures (2)

  • Figure 1: Overall framework of our model, as well as the piano assessment rubric used to construct query-answer pairs for instruction-tuning in each dataset.
  • Figure 2: Audio-text matching subjective evaluation result of three models, with errors bars indicating standard error of the mean.