Multi-granularity Interactive Attention Framework for Residual Hierarchical Pronunciation Assessment

TL;DR

The paper tackles automatic pronunciation assessment by addressing the limitations of single-granularity and unidirectional models. It introduces HIA, a residual hierarchical framework with an Interactive Attention Module that enables bidirectional interaction among phoneme, word, and utterance levels, reinforced by convolutional processing to capture local context. Key contributions include the interactive attention mechanism, residual hierarchical structure to mitigate feature forgetting, and comprehensive experiments on Speechocean762 showing state-of-the-art performance across multiple granularity and aspect metrics. The work advances CAPT by better modeling cross-granularity correlations, improving pronunciation scoring accuracy and robustness in read-aloud scenarios.

Abstract

Automatic pronunciation assessment plays a crucial role in computer-assisted pronunciation training systems. Due to the ability to perform multiple pronunciation tasks simultaneously, multi-aspect multi-granularity pronunciation assessment methods are gradually receiving more attention and achieving better performance than single-level modeling tasks. However, existing methods only consider unidirectional dependencies between adjacent granularity levels, lacking bidirectional interaction among phoneme, word, and utterance levels and thus insufficiently capturing the acoustic structural correlations. To address this issue, we propose a novel residual hierarchical interactive method, HIA for short, that enables bidirectional modeling across granularities. As the core of HIA, the Interactive Attention Module leverages an attention mechanism to achieve dynamic bidirectional interaction, effectively capturing linguistic features at each granularity while integrating correlations between different granularity levels. We also propose a residual hierarchical structure to alleviate the feature forgetting problem when modeling acoustic hierarchies. In addition, we use 1-D convolutional layers to enhance the extraction of local contextual cues at each granularity. Extensive experiments on the speechocean762 dataset show that our model is comprehensively ahead of the existing state-of-the-art methods.

Figures (4)

• Figure 1: Schematic diagram of the acoustic hierarchical structure with a sample utterance "Its good".
• Figure 2: Main architecture of HIA. HIA takes the GOP features extracted from the acoustic model and the projected canonical phoneme embeddings as input. Then, Transformer encoder is applied to encode the input to obtain the acoustic embeddings. Finally, the integrated residual hierarchical structure is used to obtain the scores at each granularity in turn.
• Figure 3: Network structure of interactive attention module. For simplicity, the residual connection and norm layers are omitted. Phn is Phoneme, Utt is Utterance.
• Figure 4: Correlation matrix of different metrics at three granularities. Thereinto, p_acc stands for phoneme-level accuracy; w_avg stands the mean value for word-level accuracy, total and stress; u_com, u_acc, u_flu, u_pros and u_tot stand for utterance-level completeness, accuracy, fluency, prosodic and total score, respectively.