Differential contributions of machine learning and statistical analysis to language and cognitive sciences

TL;DR

The paper investigates how machine learning and statistical analysis differently contribute to language and cognitive sciences by applying both paradigms to the Buckeye corpus and introducing a novel semantic relevance metric. It demonstrates that ML methods like Random Forests and SVMs achieve high predictive accuracy for word-duration categorization, while statistical approaches (LMER and GAMM) reveal robust relationships and non-linear effects among linguistic factors, with semantic relevance consistently impacting word duration. The study emphasizes the complementary strengths of data-driven prediction and interpretive inference, arguing for informed, context-dependent application of both approaches in cognitive research. These findings have practical implications for designing analyses in language production and broader cognitive science, guiding researchers on when to prioritize prediction, inference, or a hybrid strategy.

Abstract

Data-driven approaches have revolutionized scientific research, with machine learning and statistical analysis being commonly used methodologies. Despite their widespread use, these approaches differ significantly in their techniques, objectives and implementations. Few studies have systematically applied both methods to identical datasets to highlight potential differences, particularly in language and cognitive sciences. This study employs the Buckeye Speech Corpus to illustrate how machine learning and statistical analysis are applied in data-driven research to obtain distinct insights on language production. We demonstrate the theoretical differences, implementation steps, and unique objectives of each approach through a comprehensive, tutorial-like comparison. Our analysis reveals that while machine learning excels at pattern recognition and prediction, statistical methods provide deeper insights into relationships between variables. The study highlights how semantic relevance, a novel metric measuring contextual influence on target words, contributes to understanding word duration in speech. We also systematically compare the differences between regression models used in machine learning and statistical analysis, particularly focusing on the training and fitting processes. Additionally, we clarify several common misconceptions that contribute to the confusion between these two approaches. Overall, by elucidating the complementary strengths of machine learning and statistics, this research enhances our understanding of diverse data-driven strategies in language and cognitive sciences, offering researchers valuable guidance on when and how to effectively apply these approaches in different research contexts.

Figures (4)

• Figure 1: Differences between machine learning and statistics. Note: Fig. \ref{['diff']} could be aligned with Table \ref{['tab:ml_stats_differences']}, Table \ref{['table:ml_steps']}, and Table \ref{['table:gamm_steps']}, making it easier to understand it.
• Figure 2: Correlations among various factors
• Figure 3: Partial effects of a given factor on word duration. (Note: The x-axis signifies the metrics, while the y-axis delineates word duration. The plots are arranged in two rows, each containing three graphs. The top row, from left to right, displays the predictors: word length, word frequency, and CiteLength. The bottom row, also from left to right, shows: semantic relevance, phrase rate, and deletions. Each curve for an individual plot visually articulates the relation between a predictor variable and the response variable. A steeper incline on these curves demonstrates a more robust impact between the predictor and word duration, whereas gentler slopes imply a less pronounced effect. Moreover, when a curve fluctuates around zero, its effect vanishes. The p-value is smaller than 0.0001 in each plot.)
• Figure 4: The computational method for semantic relevance