Variationist: Exploring Multifaceted Variation and Bias in Written Language Data

TL;DR

This paper introduces Variationist, a highly-modular, extensible, and task-agnostic tool that fills the gap to seamlessly inspect and visualize language variation and bias across multiple variables, language units, and diverse metrics that go beyond descriptive statistics.

Abstract

Exploring and understanding language data is a fundamental stage in all areas dealing with human language. It allows NLP practitioners to uncover quality concerns and harmful biases in data before training, and helps linguists and social scientists to gain insight into language use and human behavior. Yet, there is currently a lack of a unified, customizable tool to seamlessly inspect and visualize language variation and bias across multiple variables, language units, and diverse metrics that go beyond descriptive statistics. In this paper, we introduce Variationist, a highly-modular, extensible, and task-agnostic tool that fills this gap. Variationist handles at once a potentially unlimited combination of variable types and semantics across diversity and association metrics with regards to the language unit of choice, and orchestrates the creation of up to five-dimensional interactive charts for over 30 variable type-semantics combinations. Through our case studies on computational dialectology, human label variation, and text generation, we show how Variationist enables researchers from different disciplines to effortlessly answer specific research questions or unveil undesired associations in language data. A Python library, code, documentation, and tutorials are made publicly available to the research community.

Figures (5)

• Figure 1: A high-level overview of the core elements and functionalities of Variationist. The tool computes association metrics between any unit in language and potentially unlimited variable type--semantics combinations, orchestrates the creation of interactive charts, and also supports user-defined custom components ().
• Figure 2: Example showcasing the four steps for inspecting data and visualizing results using Variationist.
• Figure 3: Example visualizations for the computational dialectology case study. All the charts have been filtered to show the use of the lexical item "ghe" across space within Italy at different granularities in terms of npw_pmi score.
• Figure 4: Example visualizations for the human label variation case study. All the charts show the npw_relevance score for the hateful class of specific lexical items across sociodemographics of annotators.
• Figure 5: Example visualizations for the text generation case study. The charts present some characteristics at the lexical level for human and ChatGPT-generated texts.