Generating novel experimental hypotheses from language models: A case study on cross-dative generalization

TL;DR

A case study where LMs are used as simulated learners to derive novel experimental hypotheses to be tested with humans to study cross-dative generalization, finding CDG to be facilitated when the first postverbal argument of the exposure context is pronominal, definite, short, and conforms to the prototypical animacy expectations of the exposure dative.

Abstract

Neural network language models (LMs) have been shown to successfully capture complex linguistic knowledge. However, their utility for understanding language acquisition is still debated. We contribute to this debate by presenting a case study where we use LMs as simulated learners to derive novel experimental hypotheses to be tested with humans. We apply this paradigm to study cross-dative generalization (CDG): productive generalization of novel verbs across dative constructions (she pilked me the ball/she pilked the ball to me)--acquisition of which is known to involve a large space of contextual features--using LMs trained on child-directed speech. We specifically ask: "what properties of the training exposure facilitate a novel verb's generalization to the (unmodeled) alternate construction?" To answer this, we systematically vary the exposure context in which a novel dative verb occurs in terms of the properties of the theme and recipient, and then analyze the LMs' usage of the novel verb in the unmodeled dative construction. We find LMs to replicate known patterns of children's CDG, as a precondition to exploring novel hypotheses. Subsequent simulations reveal a nuanced role of the features of the novel verbs' exposure context on the LMs' CDG. We find CDG to be facilitated when the first postverbal argument of the exposure context is pronominal, definite, short, and conforms to the prototypical animacy expectations of the exposure dative. These patterns are characteristic of harmonic alignment in datives, where the argument with features ranking higher on the discourse prominence scale tends to precede the other. This gives rise to a novel hypothesis that CDG is facilitated insofar as the features of the exposure context--in particular, its first postverbal argument--are harmonically aligned. We conclude by proposing future experiments that can test this hypothesis in children.

Figures (8)

• Figure 1: Overview of our methodology for investigating an LM learner's cross-dative generalization behavior for a novel dative verb (here, [pilked]).
• Figure 2: Average $\Delta$ values computed using our LM learners on naba ($N$=12) and nana ($N$=14) verbs from AO-CHILDES huebner2021using. Error bars indicate 95% CIs. Across both datives, the average $\Delta$ is significantly greater for naba verbs than it is for nana verbs ($p < .01$ for both).
• Figure 3: Asymmetric cross-dative generalization in our LM learners. Average log probability per token of the alternate forms displayed separately for different generalization sets used in this analysis. In both cases, LM learners are relatively more likely to generalize from DO to PP as compared to PP to DO ($p < .001$), analogous to the finding by conwell2007early.
• Figure 4: Average generalization set log probabilities per token for DO generalization instances for DO and PP exposures. Error bars indicate 95% CIs. Features of exposure contexts follow the ones used by arunachalam2017preschoolers. Exposure of the verb in the PP construction is more likely to facilitate usage of the verb in DO than is the exposure of the verb in the DO construction ($p < .05$), thereby replicating arunachalam2017preschoolers.
• Figure 5: Average log probabilities per token assigned to the generalization set across theme animacy configurations with other feature configurations held constant to mimic the setting used by arunachalam2017preschoolers.