Probing for the Usage of Grammatical Number

TL;DR

This paper asks how pre-trained NLP models actually use linguistic properties, proposing a usage-based probing framework that combines diagnostic, behavioral, and causal methods. It centers on grammatical number and a number-agreement task in BERT, showing that the model relies on a linear encoding that is largely orthogonal between nouns and verbs and transferred across layers, notably between layers 3 and 8, via indirect pathways rather than direct attention. By applying amnesic probes and attention pruning, the authors demonstrate that removing the encoding degrades performance, thereby linking encoding to usage, and reveal that a single linear probe may mislead about how information is used. The study advances a mechanistic view of BERT’s internal representations, highlighting POS-specific encodings and distributed information transfer as key features of how grammatical number is encoded and utilized in inference.

Abstract

A central quest of probing is to uncover how pre-trained models encode a linguistic property within their representations. An encoding, however, might be spurious-i.e., the model might not rely on it when making predictions. In this paper, we try to find encodings that the model actually uses, introducing a usage-based probing setup. We first choose a behavioral task which cannot be solved without using the linguistic property. Then, we attempt to remove the property by intervening on the model's representations. We contend that, if an encoding is used by the model, its removal should harm the performance on the chosen behavioral task. As a case study, we focus on how BERT encodes grammatical number, and on how it uses this encoding to solve the number agreement task. Experimentally, we find that BERT relies on a linear encoding of grammatical number to produce the correct behavioral output. We also find that BERT uses a separate encoding of grammatical number for nouns and verbs. Finally, we identify in which layers information about grammatical number is transferred from a noun to its head verb.

Figures (5)

• Figure 1: Linear probe performances when extracting grammatical number from BERT's intermediate layers. Layer 0 corresponds to the non-contextual representations.
• Figure 2: Cosine similarities between the learned parameter vectors of our linear probes. The matrices display similarities between different layers for a given word category (top), and across categories (bottom).
• Figure 3: Effect of our causal interventions on information recovery in subsequent layers (triangular matrices) and on the number agreement task (bar charts). Information recovery is measured at the target position by a diagnostic classifier; we display the probing accuracy drop compared to when no intervention was performed. The legend in the bar charts indicates what category the amnesic projectors have been trained on.
• Figure 4: Number agreement task performance drops after performing attention removal. The attention cut is performed on a range of layers. Rows and columns, respectively, represent the first and last intervened layer.
• Figure 5: Agreement task performance drops resulting from attention interventions, as a function of linear distance between the cue and the target. The rows represent distances (from 1 to 15) and columns represent the intervened layers. Three conditions are tested: cutting attention only at current layer (left), cutting attention starting from current layer up to the last one (middle) and from the first layer to current layer (right). The color map on the far right represent agreement scores without intervention for each linear distance.