Teaching and Critiquing Conceptualization and Operationalization in NLP

TL;DR

The paper presents a seminar designed to teach NLP students to critically examine how abstract concepts (e.g., bias, interpretability, reasoning) are conceptualized and operationalized. It articulates a cyclical, reading-heavy pedagogy using interdisciplinary critiques, paired presentations, and scaffolded feedback to foster deep critique and project design. By detailing structure, readings, and assignments, the work demonstrates how conceptual and societal critiques can shape research choices and教育 outcomes. It also discusses limitations such as scalability and the need for ethically aware, sociotechnical framing in NLP education.

Abstract

NLP researchers regularly invoke abstract concepts like "interpretability," "bias," "reasoning," and "stereotypes," without defining them. Each subfield has a shared understanding or conceptualization of what these terms mean and how we should treat them, and this shared understanding is the basis on which operational decisions are made: Datasets are built to evaluate these concepts, metrics are proposed to quantify them, and claims are made about systems. But what do they mean, what should they mean, and how should we measure them? I outline a seminar I created for students to explore these questions of conceptualization and operationalization, with an interdisciplinary reading list and an emphasis on discussion and critique.