How Do We Engage with Other Disciplines? A Framework to Study Meaningful Interdisciplinary Discourse in Scholarly Publications

TL;DR

This paper tackles the challenge of measuring meaningful interdisciplinarity in scholarly publications, arguing that traditional diversity-based metrics miss how deeply authors engage with out-of-discipline ideas. It introduces a novel, inductively developed citation purpose taxonomy and an annotation framework, applied to NLP+CSS publications, to quantify the depth of interdisciplinary engagement. Through an annotated dataset of 369 citations and a detailed engagement analysis, the work reveals that surface-level engagement is prevalent and that semantic similarity alone cannot infer engagement depth. The study also explores automated classification approaches, finding current models insufficient for reliable large-scale use, thereby highlighting the need for more annotated data and stronger context-aware methods to enable scalable interdisciplinarity analysis with practical impact for researchers and evaluators.

Abstract

With the rising popularity of interdisciplinary work and increasing institutional incentives in this direction, there is a growing need to understand how resulting publications incorporate ideas from multiple disciplines. Existing computational approaches, such as affiliation diversity, keywords, and citation patterns, do not account for how individual citations are used to advance the citing work. Although, in line with addressing this gap, prior studies have proposed taxonomies to classify citation purpose, these frameworks are not well-suited to interdisciplinary research and do not provide quantitative measures of citation engagement quality. To address these limitations, we propose a framework for the evaluation of citation engagement in interdisciplinary Natural Language Processing (NLP) publications. Our approach introduces a citation purpose taxonomy tailored to interdisciplinary work, supported by an annotation study. We demonstrate the utility of this framework through a thorough analysis of publications at the intersection of NLP and Computational Social Science.

Figures (12)

• Figure 1: Framework for evaluating a publication's level of engagement with citing work.
• Figure 2: Dataset of NLP+CSS articles across years and venues
• Figure 3: Types of agreements and disagreements among annotators.
• Figure 4: Correlation between citation purpose and context relatedness.
• Figure 5: Correlation between citation section and purpose.