Testing learning hypotheses using neural networks by manipulating learning data

TL;DR

The study investigates how English speakers learn exceptions to passivization by leveraging neural network language models as acquisition theories and manipulating their training data to test competing sources of indirect evidence. It shows that models trained on human-scale data can approximate human judgments of passivization exceptions and that reducing a verb’s frequency in the passive strongly shifts its passivizability, while changing the verb’s semantics has more limited impact. The work demonstrates a practical method to causally link input features to learning outcomes, highlighting frequency-driven signals as a plausible driver for passivization restrictions and offering a nuanced view of gradient, class-, and verb-level effects. Together, these results inform our understanding of how humans might learn complex syntactic generalizations from indirect evidence and illustrate a rigorous data-centric approach for probing language acquisition mechanisms.

Abstract

Although passivization is productive in English, it is not completely general -- some exceptions exist (e.g. *One hour was lasted by the meeting). How do English speakers learn these exceptions to an otherwise general pattern? Using neural network language models as theories of acquisition, we explore the sources of indirect evidence that a learner can leverage to learn whether a verb can passivize. We first characterize English speakers' judgments of exceptions to the passive, confirming that speakers find some verbs more passivizable than others. We then show that a neural network language model can learn restrictions to the passive that are similar to those displayed by humans, suggesting that evidence for these exceptions is available in the linguistic input. We test the causal role of two hypotheses for how the language model learns these restrictions by training models on modified training corpora, which we create by altering the existing training corpora to remove features of the input implicated by each hypothesis. We find that while the frequency with which a verb appears in the passive significantly affects its passivizability, the semantics of the verb does not. This study highlight the utility of altering a language model's training data for answering questions where complete control over a learner's input is vital.

Figures (10)

• Figure 1: Example survey question
• Figure 2: Passive drop in human acceptability judgments of active and passive sentences by verb --- The steeper the downward gradient between active and passive conditions, the larger the passive drop. Error bars indicate bootstrapped 95% confidence intervals.
• Figure 3: Accuracy of acceptability judgments on BLiMP --- Our models perform marginally worse than a GPT-2 model trained on more data, and better than an LSTM model. Dashed lines indicate chance-level accuracy. GPT-2 and LSTM results obtained from warstadt2020.
• Figure 4: Passive drop in humans vs. neural network language models --- Our models approximately predict variable amounts of passive drop equivalent to human judgments. Each point represents the average passive drop of a verb in five sentence frames scored by five models. Horizontal error bars indicate bootstrapped 95% confidence intervals over participants and sentence frames; vertical error bars indicate bootstrapped 95% confidence intervals over the different models and sentence frames.
• Figure 5: Frequency of occurrence of mutating and target verbs in the original corpus --- Duration verbs tended to occur relatively infrequently in the passive compared to agent-patient verbs.