Open Sesame: Getting Inside BERT's Linguistic Knowledge

TL;DR

The paper investigates how BERT encodes syntactic structure, distinguishing positional/linear information from hierarchical information. It combines a poverty-of-the-stimulus diagnostic framework with a novel attention-based confusion score to quantify how self-attention encodes dependencies like subject–verb agreement and reflexive anaphora. Across two BERT variants, the study shows that lower layers preserve positional cues while higher layers encode progressively more abstract hierarchical features, though human-like sensitivity to certain dependencies remains imperfect. The results demonstrate that BERT builds increasingly abstract syntactic representations through attention mechanisms, which helps explain its strong performance on structure-sensitive NLP tasks while highlighting areas needing further exploration. Overall, the work provides a concrete methodology for dissecting linguistic knowledge in transformer models and contributes nuanced insights into how hierarchical structure is instantiated in self-attention layers.

Abstract

How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT's representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT's representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT's representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.

Figures (5)

• Figure 1: Layerwise accuracy of diagnostic classifiers on the generalization set of the main auxiliary task.
• Figure 2: Layerwise accuracy of diagnostic classifiers on the compound noun generalization set of the subject noun task.
• Figure 3: Layerwise accuracy of diagnostic classifiers on the possessive generalization set of the subject noun task.
• Figure 4: Layerwise accuracy of diagnostic classifiers on the generalization set of the $n^{th}$ token task, for the bbu model only. Note that each line corresponds to a particular layer's embeddings as we vary $2 \leq n \leq 9$. pE denotes pre-embeddings and pE -- pos denotes pre-embeddings without the positional component.
• Figure 5: Layerwise confusion scores for each reflexive anaphora condition listed in Table \ref{['tb:dataset_result_attention']}. Conditions R1 to R4 have one distractor noun phrase, but conditions R5 to R8 have two distractor noun phrases.