Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates

Abstract

Solidarity is a crucial concept to understand social relations in societies. In this paper, we explore fine-grained solidarity frames to study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. Using 2,864 manually annotated text snippets (with a cost exceeding 18k Euro), we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4. We find that GPT-4 outperforms other LLMs, approaching human annotation quality. Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years and find that solidarity with migrants outweighs anti-solidarity but that frequencies and solidarity types shift over time. Most importantly, group-based notions of (anti-)solidarity fade in favor of compassionate solidarity, focusing on the vulnerability of migrant groups, and exchange-based anti-solidarity, focusing on the lack of (economic) contribution. Our study highlights the interplay of historical events, socio-economic needs, and political ideologies in shaping migration discourse and social cohesion. We also show that powerful LLMs, if carefully prompted, can be cost-effective alternatives to human annotation for hard social scientific tasks.

Figures (17)

• Figure 1: Annotation scheme based on thijssen2012mechanical. The scheme categorizes statements into solidarity, anti-solidarity, mixed, and none (high-level). At the fine-grained level, solidarity and anti-solidarity are further divided into group-based, exchange-based, compassionate, and empathic subtypes.
• Figure 2: Number of instances in the Woman and Migrant dataset in each year. Fig. \ref{['fig:relative-frequency-records']} in the Appendix illustrates the relative frequency of instances in both datasets.
• Figure 3: Fig. \ref{['subfig:annotation-confusion-matrix-highlevel']}shows the confusion matrix between human annotators; Fig. \ref{['subfig:confusion-matrix-highlevel']} shows agreement between the best model and human annotators on a test set from one of the three splits. The former is aggregated over all pairwise comparisons of annotators, thus the matrix is symmetric.
• Figure 4: Distribution of instances in the human annotated dataset across time and target groups. See Fig. \ref{['fig:instances-distribution-categories']} in the Appendix for the plots for each group separately; and Table \ref{['tab:actual-instances-distribution-categories']} for the actual numbers of instances in the human annotated dataset.
• Figure 5: Fig. \ref{['fig:solidarity-per-decade']} shows the fraction of solidarity, anti-solidarity, and mixed stance towards migrants. Fig. \ref{['fig:solidarity-per-decade']} shows the fraction of solidarity (left) and anti-solidarity (right) subtypes according to GPT-4, where percentages represent the proportion of each subtype relative to the total counts of solidarity or anti-solidarity labels per decade. Grey shaded areas from 1933 to 1949 indicate sparse data during the NS dictatorship and post-war period.