Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HuPER: A Human-Inspired Framework for Phonetic Perception

Chenxu Guo, Jiachen Lian, Yisi Liu, Baihe Huang, Shriyaa Narayanan, Cheol Jun Cho, Gopala Anumanchipalli

TL;DR

HuPER addresses the bottleneck of phonetic perception by proposing a human-inspired framework that performs adaptive, multi-path inference over acoustic evidence and linguistic knowledge. It decomposes perception into a bottom-up HuPER-Recognizer, a lexical/top-down HuPER-Perceiver, a Dysfluent WFST for constraint-based refinement, and a HuPER-Scheduler that routing decisions based on evidence quality. With a data-efficient training regimen (only ~100 hours) and a novel self-learning DRRC objective, HuPER achieves state-of-the-art PFER on English benchmarks and strong zero-shot transfer to 95 languages, while enabling robust, reference-guided decoding under degraded conditions. The work demonstrates that combining structured constraints, dynamic control, and explicit perceptual mechanisms yields improved phonetic representations and robustness, with open-source resources facilitating broader adoption and future extensions to more expressive perceptual models.

Abstract

We propose HuPER, a human-inspired framework that models phonetic perception as adaptive inference over acoustic-phonetics evidence and linguistic knowledge. With only 100 hours of training data, HuPER achieves state-of-the-art phonetic error rates on five English benchmarks and strong zero-shot transfer to 95 unseen languages. HuPER is also the first framework to enable adaptive, multi-path phonetic perception under diverse acoustic conditions. All training data, models, and code are open-sourced. Code and demo avaliable at https://github.com/HuPER29/HuPER.

HuPER: A Human-Inspired Framework for Phonetic Perception

TL;DR

HuPER addresses the bottleneck of phonetic perception by proposing a human-inspired framework that performs adaptive, multi-path inference over acoustic evidence and linguistic knowledge. It decomposes perception into a bottom-up HuPER-Recognizer, a lexical/top-down HuPER-Perceiver, a Dysfluent WFST for constraint-based refinement, and a HuPER-Scheduler that routing decisions based on evidence quality. With a data-efficient training regimen (only ~100 hours) and a novel self-learning DRRC objective, HuPER achieves state-of-the-art PFER on English benchmarks and strong zero-shot transfer to 95 languages, while enabling robust, reference-guided decoding under degraded conditions. The work demonstrates that combining structured constraints, dynamic control, and explicit perceptual mechanisms yields improved phonetic representations and robustness, with open-source resources facilitating broader adoption and future extensions to more expressive perceptual models.

Abstract

We propose HuPER, a human-inspired framework that models phonetic perception as adaptive inference over acoustic-phonetics evidence and linguistic knowledge. With only 100 hours of training data, HuPER achieves state-of-the-art phonetic error rates on five English benchmarks and strong zero-shot transfer to 95 unseen languages. HuPER is also the first framework to enable adaptive, multi-path phonetic perception under diverse acoustic conditions. All training data, models, and code are open-sourced. Code and demo avaliable at https://github.com/HuPER29/HuPER.
Paper Structure (72 sections, 2 theorems, 45 equations, 8 figures, 8 tables, 2 algorithms)

This paper contains 72 sections, 2 theorems, 45 equations, 8 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. HuPER overview
  3. HuPER-Recognizer
  4. Task definition and outputs.
  5. HuPER self-learning strategy.
  6. DRRC perspective.
  7. HuPER-Perceiver
  8. Acoustic evidence as a phone lattice.
  9. Imposing lexical and linguistic constraints.
  10. Search via WFST composition.
  11. HuPER-Scheduler
  12. Quantifying Evidence Distortion.
  13. Dysfluent WFST constraint $\mathcal{H}(\cdot)$.
  14. Experimental setup
  15. Phone recognition
  16. ...and 57 more sections

Key Result

Theorem 3.1

For any measurable $g:\mathcal{Z}\times\{1,\dots,K\}\to(0,1]$, define Let $\hat{g}$ be a cross-fitted estimator clipped to $[\varepsilon,1]$ and define Then $\mathbb{E}[\ell_\theta(C_g(W),X)] = R(\theta)$ provided either $\textbf{(G): } g = g^*$ or $\textbf{(Y): } \hat{Y} = Y$. Furthermore, $\hat{R}_n(\theta)\to R(\theta)$ in probability provided either:

Figures (8)

  • Figure 1: HuPER achieves highly data-efficient phonetic transcription on English variation benchmarks.
  • Figure 2: HuPER overview: evidence-controlled multi-path speech perception. Left: HuPER-Recognizer (Encoder/Decoder; mapped to STG) converts the speech signal into acoustic--phonetic features (orange) and phone posteriors (green). HuPER-Scheduler (mapped to IFG) monitors evidence strength and selects an inference route. Right: (a) Oral reading transcription: with an external prompt/reference, a Dysfluent WFST applies explicit top-down constraints to produce the transcript. (b) Normal spontaneous speech: when evidence is strong, the system trusts bottom-up phone evidence and outputs directly. (c) Less intelligible speech: when evidence is weak, HuPER-Perceiver combines phone evidence with a lexical prior to form word hypotheses (blue), which are then refined by the Dysfluent WFST conf/interspeech/GuoLZZLYPDEVMBW25.
  • Figure 3: HuPER-Recognizer self-learning pipeline. The training procedure consists of four stages. (1) An initial phone recognizer is trained on a small human-annotated corpus (TIMIT) to produce acoustic phone predictions. (2) The recognizer is applied to a large transcript-only corpus (LibriSpeech) to generate teacher pseudo phones from speech, while a G2P system produces canonical phoneme sequences from text. (3) A Corrector model learns edit operations (keep, delete, substitute, insert) that transform canonical G2P phones into acoustically grounded phone proxies, using both speech tokens and G2P phones as input. (4) The recognizer is retrained on the large corpus using corrected pseudo phone labels, yielding a more robust and language-generalizable phone recognizer.
  • Figure 4: Zero-shot multilingual phone recognition on VoxAngeles (95 languages). Per-language phone error rate (PFER, $\downarrow$) for HuPER-Recognizer (trained only on English) and an English-only public phoneme model Wav2Vec2-en (fine-tuned on G2P-generated phoneme labels from LJSpeech). HuPER improves on the majority of languages and reduces the macro-average PFER from 0.35 to 0.19.
  • Figure 5: Centroid RSA to acoustic--phonetic geometry. Spearman correlation between pairwise phone-centroid cosine distances and PanPhon distinctive-feature distances across layers. Higher values indicate stronger acoustic--phonetic organization.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Theorem 3.1: Informal version of \ref{['thm:dr']}
  • Theorem 2.5: Proxy-baseline AIPW corrector is doubly robust
  • proof