Ethical Concern Identification in NLP: A Corpus of ACL Anthology Ethics Statements

TL;DR

The introduction of EthiCon, a corpus of 1,580 ethical concern statements extracted from scientific papers published in the ACL Anthology, and promising results in automating the concern identification process are shown.

Abstract

What ethical concerns, if any, do LLM researchers have? We introduce EthiCon, a corpus of 1,580 ethical concern statements extracted from scientific papers published in the ACL Anthology. We extract ethical concern keywords from the statements and show promising results in automating the concern identification process. Through a survey, we compare the ethical concerns of the corpus to the concerns listed by the general public and professionals in the field. Finally, we compare our retrieved ethical concerns with existing taxonomies pointing to gaps and future research directions.

Figures (12)

• Figure 1: Visualizing top 60 concerns in ACL ethics statements, reflecting term frequencies.
• Figure 2: Examples from the identified categories of ethical concern statements.
• Figure 3: Distribution of categories of the 1,100 ethics statements in the EthiCon dataset from ACL 2023.
• Figure 4: The five most frequent ethical concerns in the statements from ACL 2022--3 anthologies.
• Figure 5: Comparing concerns between professional and regular users (1--5 Likert scale, with 1 'Not worried at all'). The radar plot illustrates the average levels of concern across the participants per category/question.