Spoken Grammar Assessment Using LLM

TL;DR

This paper proposes a novel end-to-end SLA system to assess language grammar from spoken utterances thus making WLA systems redundant and makes the assessment largely unteachable by employing a large language model (LLM) to bring in variations in the test.

Abstract

Spoken language assessment (SLA) systems restrict themselves to evaluating the pronunciation and oral fluency of a speaker by analysing the read and spontaneous spoken utterances respectively. The assessment of language grammar or vocabulary is relegated to written language assessment (WLA) systems. Most WLA systems present a set of sentences from a curated finite-size database of sentences thereby making it possible to anticipate the test questions and train oneself. In this paper, we propose a novel end-to-end SLA system to assess language grammar from spoken utterances thus making WLA systems redundant; additionally, we make the assessment largely unteachable by employing a large language model (LLM) to bring in variations in the test. We further demonstrate that a hybrid automatic speech recognition (ASR) with a custom-built language model outperforms the state-of-the-art ASR engine for spoken grammar assessment.

Figures (5)

• Figure 1: End to End System for SLA. We only look at the grammar of spoken language.
• Figure 2: A sample $P$ generated using an LLM along with $P_d$ used to display and $P_g$ used for grammar assessment.
• Figure 3: Sample sentence (a) and expected variations (b).
• Figure 4: $1$-shot learning prompting to generate new $P$.
• Figure 5: Paragraph's generated by prompting ChatGPT.