The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?

TL;DR

It is suggested that the potential of LLMs in musicology requires musicology driven research that can specialized LLMs by including accurate and reliable domain knowledge.

Abstract

In this work, we explore the use and reliability of Large Language Models (LLMs) in musicology. From a discussion with experts and students, we assess the current acceptance and concerns regarding this, nowadays ubiquitous, technology. We aim to go one step further, proposing a semi-automatic method to create an initial benchmark using retrieval-augmented generation models and multiple-choice question generation, validated by human experts. Our evaluation on 400 human-validated questions shows that current vanilla LLMs are less reliable than retrieval augmented generation from music dictionaries. This paper suggests that the potential of LLMs in musicology requires musicology driven research that can specialized LLMs by including accurate and reliable domain knowledge.

Figures (5)

• Figure 1: Fictitious interaction illustrating why LLMs' hallucinations might prevent musicologists' trust.
• Figure 2: Survey's answers about the usage of LLMs in general (left) and on music topics (right)
• Figure 3: Survey's answers about the usage of LLMs in general (left) and on music topics (right).
• Figure 4: Survey answers about the revolutionary impact (right) and potential consequences (left) of LLMs.
• Figure 5: Language chain for generating the multi-choice questions.