Investigating grammatical abstraction in language models using few-shot learning of novel noun gender

TL;DR

The study probes whether small, uni-directional language models can form abstract grammatical gender representations for novel French nouns by updating only the embedding layer in a few-shot learning setting. It employs a controlled word-learning paradigm with one-shot or few-shot updates across multiple agreement contexts and compares an LSTM and a decoder-only Transformer, showing both can generalize gender from minimal examples, albeit with a masculine bias. A parallel human experiment reveals a similar masculine bias among native speakers, highlighting shared challenges in one-shot gender induction. The results suggest that grammatical gender information can be encoded in word embeddings to enable cross-context agreement, but the exact mechanistic basis remains to be clarified and warrants further mechanistic and cross-language investigation.

Abstract

Humans can learn a new word and infer its grammatical properties from very few examples. They have an abstract notion of linguistic properties like grammatical gender and agreement rules that can be applied to novel syntactic contexts and words. Drawing inspiration from psycholinguistics, we conduct a noun learning experiment to assess whether an LSTM and a decoder-only transformer can achieve human-like abstraction of grammatical gender in French. Language models were tasked with learning the gender of a novel noun embedding from a few examples in one grammatical agreement context and predicting agreement in another, unseen context. We find that both language models effectively generalise novel noun gender from one to two learning examples and apply the learnt gender across agreement contexts, albeit with a bias for the masculine gender category. Importantly, the few-shot updates were only applied to the embedding layers, demonstrating that models encode sufficient gender information within the word embedding space. While the generalisation behaviour of models suggests that they represent grammatical gender as an abstract category, like humans, further work is needed to explore the details of how exactly this is implemented. For a comparative perspective with human behaviour, we conducted an analogous one-shot novel noun gender learning experiment, which revealed that native French speakers, like language models, also exhibited a masculine gender bias and are not excellent one-shot learners either.

Figures (10)

• Figure 1: LSTM (top) and transformer (bottom) prediction accuracies of gender agreement with existing French nouns that appear in training data, across three agreement tests. Error bars are 95% confidence intervals across sentences.
• Figure 2: LSTM performance on gender agreement tests before learning, corresponding to zero sentences learnt, and after few-shot learning with 1, 2, 3, 5 and 10 sentences. The dark orange lines plot average prediction accuracy after learning from feminine training sentences, while the blue lines correspond to learning masculine sentences. The faded lines indicate the individual performance of 20 novel nouns. The left $y$-axis shows the prediction accuracy for feminine gender, while the right $y$-axis displays masculine gender accuracy such that $100\%$ accuracy for feminine gender corresponds to $0\%$ for masculine gender. Error bars of 95% bootstrapped confidence intervals are too small to be seen.
• Figure 3: Transformer performance on gender agreement tests before and after few-shot learning. See Figure \ref{['fig:lstm_learning']} caption for details on layout, axes and content of graphs.
• Figure 4: Transformer: Top ten tokens by percentage of weight change to embedding layer after few-shot learning updates with 1-10 sentences. Note that the colour scales representing percentage change are different in each panel.
• Figure 5: LSTM: Top 10 tokens by percentage of weight change to embedding layer after few-shot learning updates. Each panel shows weight changes for 1-10 learning constructions indicating feminine or masculine noun novel gender with sentence constructions from each test condition: A/B article-noun (top), C noun-adjective (mid) D noun-participle (bottom). See Table \ref{['tab:tests']} for learning constructions. Top tokens include the novel noun highlighted in green and other expected words from the learning examples. Note that the percentage change color scales are different in each panel.