Overview of PerpectiveArg2024: The First Shared Task on Perspective Argument Retrieval

TL;DR

This paper introduces the first Shared Task on Perspective Argument Retrieval, proposing three perspectivism scenarios and a multilingual dataset that encodes socio-cultural variables to study personalized argument retrieval. It defines a dual-evaluation framework focusing on relevance and diversity, and reports results from six participating teams across three test cycles, highlighting substantial challenges in encoding perspectivism and persistent biases toward majority groups. The study shows that incorporating socio-cultural variables is difficult without explicit signals and that temporal shifts between data sources degrade performance, underscoring the need for better signals and robust evaluation. The work lays a foundation for research into perspectivism-aware retrieval and personalization to reduce polarization while ensuring fairness and broad representation.

Abstract

Argument retrieval is the task of finding relevant arguments for a given query. While existing approaches rely solely on the semantic alignment of queries and arguments, this first shared task on perspective argument retrieval incorporates perspectives during retrieval, accounting for latent influences in argumentation. We present a novel multilingual dataset covering demographic and socio-cultural (socio) variables, such as age, gender, and political attitude, representing minority and majority groups in society. We distinguish between three scenarios to explore how retrieval systems consider explicitly (in both query and corpus) and implicitly (only in query) formulated perspectives. This paper provides an overview of this shared task and summarizes the results of the six submitted systems. We find substantial challenges in incorporating perspectivism, especially when aiming for personalization based solely on the text of arguments without explicitly providing socio profiles. Moreover, retrieval systems tend to be biased towards the majority group but partially mitigate bias for the female gender. While we bootstrap perspective argument retrieval, further research is essential to optimize retrieval systems to facilitate personalization and reduce polarization.

Figures (20)

• Figure 1: This example entry shows which information we consider for this shared task. First, we incorporate the semantic information as the text of queries and arguments. Secondly, we use the demographic and socio-cultural properties (perspective) of argument authors or users, including age, gender, or political attitude.
• Figure 2: Examples of query and a relevant argument for the three scenarios: (1) no perspectivism without socio variables; (3) explicit perspectivism with socio variable in query and argument; (3) implicit perspectivism with socio variable only in the query.
• Figure 3: Distribution of the politicians' different demographic and socio-cultural variables: important political issues, political attitude, residence, gender, age (binned), civil status, and denomination. Note, that one person can have more than one important political issue.
• Figure 4: Overview of train, dev, and test argument corpora ($C$) and queries $q$ for the three evaluation cycles dataset (2019, 2023, surprise)
• Figure 5: Performance overview regarding the four measured metrics and their relation. The color indicates the specific scenario.