PhonologyBench: Evaluating Phonological Skills of Large Language Models

TL;DR

PhonologyBench tackles a gap in LLM evaluation by systematically benchmarking three English phonological skills—grapheme-to-phoneme conversion, syllable counting, and rhyme word generation—using text-only trained models. The study analyzes six LLMs (open and closed source) and investigates the influence of word frequency, tokenization, and sentence complexity, revealing that while models perform better than chance, they remain substantially below human performance, with notable gaps in syllable counting and rhyme generation. Key findings include that no single model dominates across all tasks and that tokenization and data exposure critically shape phonological capability, suggesting targeted data and task-aligned model selection for downstream phonology-sensitive applications. The benchmark and results underscore the practical importance of incorporating phonological evaluation in LLM development, and point to future work in phonetic augmentation, multilingual phonology, and dialectal variation to improve real-world performance.

Abstract

Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.

Figures (7)

• Figure 1: Overview of tasks comprising PhonologyBench. a) Grapheme-to-Phoneme conversion translates written language into phonetic script b) Syllable Counting enumerates the syllables in a sentence c) Rhyme Word Generation identifies rhyming counterparts for a given word. Each task is presented with an example and various downstream tasks.
• Figure 2: Samples from all three tasks in PhonologyBench
• Figure 3: Performance of the 6 evaluated models on PhonologyBench. We consider human performance on Rhyme Word Generation and Syllable Counting as the baseline. For G2P, we consider a dictionary look-up based approach as the baseline.
• Figure 4: Performance of the 4 models on syllable counting at varying sentence lengths.
• Figure 5: We provide responses of GPT-4, Claude-3-Sonnet, LLaMA-2-13B-Chat and Mistral-Instruct-7B on two samples from the Syllable Counting task.