Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

Abstract

Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.

Figures (16)

• Figure 1: Example of Change of state verbs and Object drop verbs alternation and morphological marking across languages. The verb is highlighted in bold while colours code semantic roles (agents, patients), morphological marking on the verb (si in Italian) and morphological case marking on the argument nominative and accusative case.
• Figure 2: Simplified example of a BLM template.
• Figure 3: Example of template of type B
• Figure 4: Example of template of type C
• Figure 5: A simplified example of MaxLex of the template of Figure \ref{['fig:BLM-overview']}. The sentences follow the same syntax, but the words vary in each sentence.