Probing Internal Representations of Multi-Word Verbs in Large Language Models
Hassane Kissane, Achim Schilling, Patrick Krauss
TL;DR
This paper examines how large language models encode two types of multi-word verbs—phrasal and prepositional verbs—by applying probing classifiers to BERT representations at token and sentence levels. Using LR and linear SVM across 12 layers and measuring cluster separability with Generalized Discrimination Value (GDV), the study finds that middle layers contain the most discriminative information, with token-level accuracy reaching up to $0.99$ and sentence-level accuracy peaking in the $0.84$–$0.85$ range. GDV analyses indicate weak linear separability between verb types ($ ext{GDV}$ values around $-0.049$ at best), suggesting that the representations are primarily non-linear and distributed. These results align with usage-based and constructionist perspectives, highlighting that probing accuracy alone does not fully reveal internal model representations and that multiple analytic methods are needed to capture the richness of linguistic structure in LLMs.
Abstract
This study investigates the internal representations of verb-particle combinations, called multi-word verbs, within transformer-based large language models (LLMs), specifically examining how these models capture lexical and syntactic properties at different neural network layers. Using the BERT architecture, we analyze the representations of its layers for two different verb-particle constructions: phrasal verbs like 'give up' and prepositional verbs like 'look at'. Our methodology includes training probing classifiers on the internal representations to classify these categories at both word and sentence levels. The results indicate that the model's middle layers achieve the highest classification accuracies. To further analyze the nature of these distinctions, we conduct a data separability test using the Generalized Discrimination Value (GDV). While GDV results show weak linear separability between the two verb types, probing classifiers still achieve high accuracy, suggesting that representations of these linguistic categories may be non-linearly separable. This aligns with previous research indicating that linguistic distinctions in neural networks are not always encoded in a linearly separable manner. These findings computationally support usage-based claims on the representation of verb-particle constructions and highlight the complex interaction between neural network architectures and linguistic structures.