Spoken Language Intelligence of Large Language Models for Language Learning

TL;DR

The paper investigates the Spoken Language Intelligence (SLI) of large language models for language learning by introducing the SLIQ-LL dataset, a benchmark of 445 MCQs on phonetics, phonology, and second language acquisition. It systematically analyzes prompting strategies (zero-shot, few-shot, chain-of-thought), in-domain exemplars, self-consistency, and tool augmentation across 20 models, revealing that larger models achieve higher accuracy and stability, especially on knowledge-structure items, while reasoning in real-world contexts remains challenging. Key findings show that in-domain prompts substantially boost performance, self-consistency helps some models, and external tools offer limited but notable benefits; GPT-4 achieves top performance, with notable gaps remaining in practical reasoning and conversational SLI. The work highlights the potential and limits of LLM-based language-learning assistants and suggests directions for multimodal evaluation and knowledge-repository augmentation to bring spoken-language tutoring closer to real-world classroom needs.

Abstract

People have long hoped for a conversational system that can assist in real-life situations, and recent progress on large language models (LLMs) is bringing this idea closer to reality. While LLMs are often impressive in performance, their efficacy in real-world scenarios that demand expert knowledge remains unclear. LLMs are believed to hold the most potential and value in education, especially in the development of Artificial intelligence (AI) based virtual teachers capable of facilitating language learning. Our focus is centered on evaluating the efficacy of LLMs in the realm of education, specifically in the areas of spoken language learning which encompass phonetics, phonology, and second language acquisition. We introduce a new multiple-choice question dataset to evaluate the effectiveness of LLMs in the aforementioned scenarios, including understanding and application of spoken language knowledge. In addition, we investigate the influence of various prompting techniques such as zero- and few-shot method (prepending the question with question-answer exemplars), chain-of-thought (CoT, think step-by-step), in-domain exampler and external tools (Google, Wikipedia). We conducted large-scale evaluation on popular LLMs (20 distinct models) using these methods. We achieved significant performance improvements compared to the zero-shot baseline in the practical questions reasoning (GPT-3.5, 49.1% -> 63.1%; LLaMA2-70B-Chat, 42.2% -> 48.6%). We found that models of different sizes have good understanding of concepts in phonetics, phonology, and second language acquisition, but show limitations in reasoning for real-world problems. Additionally, we also explore preliminary findings on conversational communication.

Figures (9)

• Figure 1: An example of answering a SLIQ-LL Application Question using zero-shot CoT prompting “Let’s think step by step”kojima2022large.
• Figure 2: Left: Distribution of problem types in the Application Questions subset. We labeled each question based on the problem and all corresponding options. Each question may have one or multiple types. Right: Distribution of four answer options. Overall refers to the entire dataset. We report the distribution of answer options for different problem types in the Application Questions subset.
• Figure 3: Overall performance on the SLIQ-LL dataset. We report the best results of each model among four different prompting methods. The results in the figure are the raw results minus 25% (random selection level).
• Figure 4: The distribution of performance on two subsets across different model sizes.
• Figure 5: The accuracy distribution across different question types in the Application Questions subset.