Classification errors distort findings in automated speech processing: examples and solutions from child-development research

TL;DR

This paper examines how errors from automated voice-type classifiers distort downstream research in child language development using long-form audio. It introduces a flexible Bayesian calibration framework that jointly models the child speech behavior and the classifier's error process, enabling unbiased estimation of vocalization quantities and their relationships. Through calibration on manually annotated data and validation via simulations, it shows that classification errors can substantially bias direct measurements, associations, and developmental effects, though Bayesian calibration can mitigate many biases by widening credible intervals to reflect uncertainty. The work provides practical recommendations for researchers and a Python toolbox to simulate classifier impacts, thereby improving the reliability of inference in studies relying on automated speech processing. Overall, it highlights the need for measurement-error-aware analyses in wearable-sensor studies and offers a concrete path to recover more trustworthy conclusions from imperfect classifiers.

Abstract

With the advent of wearable recorders, scientists are increasingly turning to automated methods of analysis of audio and video data in order to measure children's experience, behavior, and outcomes, with a sizable literature employing long-form audio-recordings to study language acquisition. While numerous articles report on the accuracy and reliability of the most popular automated classifiers, less has been written on the downstream effects of classification errors on measurements and statistical inferences (e.g., the estimate of correlations and effect sizes in regressions). This paper's main contributions are drawing attention to downstream effects of confusion errors, and providing an approach to measure and potentially recover from these errors. Specifically, we use a Bayesian approach to study the effects of algorithmic errors on key scientific questions, including the effect of siblings on children's language experience and the association between children's production and their input. By fitting a joint model of speech behavior and algorithm behavior on real and simulated data, we show that classification errors can significantly distort estimates for both the most commonly used \gls{lena}, and a slightly more accurate open-source alternative (the Voice Type Classifier from the ACLEW system). We further show that a Bayesian calibration approach for recovering unbiased estimates of effect sizes can be effective and insightful, but does not provide a fool-proof solution.

Figures (25)

• Figure 1: 30-second sample of a daylong recording annotated by a human expert and two algorithms: , and . CHI refers to the child wearing the recording device; OCH refers to other children; FEM and MAL refer to female and male adults. A segment of speech is referred to as a "vocalization" (for instance, the expert found two female adult vocalizations in this portion of audio, but found none). Vocalization counts are shown to the right.
• Figure 2: The quantity of speech attributed to each speaker ("CHI", "OCH", "FEM", "MAL", i.e. the key child, other children, female adults, and male adults) in each recording by an algorithm only indirectly reflect the true quantities. In reality, speaker classification errors can distort measurements and create spurious correlations in the quantities of speech attributed to each speaker. (\ref{['fig:example_fem_prop']}) Measurements of speech quantities. The nature of the input to children may be misrepresented as a result of classification errors. For instance, the proportion of female adult speech can be distorted due to incorrect inferences about the speaker's type and gender. (\ref{['fig:example_associations']}) Associations between speakers. An increase in female adult speech may trigger an increase in detected amounts of both female adult (black arrow) and child speech (red arrow), and vice-versa, creating the appearance of an association between the two speakers. (\ref{['fig:example_siblings']}) Effect of independent variables on speech quantities. Spurious associations can also affect inferences about the effect of independent variables on speech behavior. For example, we might draw incorrect conclusions about the existence and direction of an effect of siblings on the quantity of speech received from adults (dashed lines) if speech from siblings is incorrectly classified as adult speech (then, children with siblings might falsely appear to receive more input from adults).
• Figure 3: Correlations between speakers' vocalization counts in $6638 \times 15$s audio clips according to human, , and annotations. Estimates are generally inconsistent across the three.
• Figure 4: Model of speech behavior. Observed variables (vocalization counts for each speaker and recording, child age, and siblings number) are shown in blue, latent variables in white. Indices $k$ designate recordings, and $c$ designates a child.$v_k^{\text{recs}}$ is the vocalization count of each speaker class in each recording. Variables $\mu$ represent the expected speech rates per speaker at each level (population, corpus, and child). $\alpha_{c}^{\text{dev}}$ is the random effect of age on the children's output (which is assumed to be distributed around a mean value $\alpha_{\text{dev}}$). It is also assumed that the expected quantity of adult speech at the child level has a long-term effect on children's speech ($\beta^{\text{dev}}$), which interacts with age.
• Figure 5: Combined model of speech behavior and of the algorithm behavior. Compared to Figure \ref{['fig:partial_model']}, the actual quantity of vocalizations $(v^{\text{recs}})$ is treated as latent variables. Colored arrows represent the effect of real vocalizations from each speaker (e.g. CHI, in blue) on the amount of vocalizations attributed to each speaker by the algorithm $(n^{\text{recs}})$. The unobserved confusion rates $\lambda_{kij}$ represent the rate at which vocalizations from a speaker $i$ are detected and attributed to a speaker $j$ in recording $k$. The distribution of $\lambda_{kij}$, parameterized by $\mu_{ij}$ and $\alpha_{ij}$, is learned via calibration data (for which both the true counts $n^{\text{clips}}$ and the algorithmic counts $v^{\text{clips}}$ are known).