Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts

TL;DR

This work formalizes a cross-domain schema for NLP research framings and develops an automated three-stage framework to infer framings from papers: extract epistemic elements, link them via interpretable rules to candidate framings, and refine predictions with an LLM. Evaluations on automated fact-checking and hate speech detection show the approach outperforms a strong LLM baseline and excels at identifying underspecified framings, revealing trends toward more scientific curiosity and greater justification for human-in-the-loop systems. The authors demonstrate scalability by applying the framework to over a hundred AFC papers, uncovering shifts toward human-assisted fact-checking and richer justification generation. Overall, the work enables scalable, transparent analysis of how NLP artefacts are framed and used, informing responsible research practices and framing transparency.

Abstract

Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated fact-checking using an existing dataset, and hate speech detection for which we annotate a new dataset-achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in vague or underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human fact-checkers rather than pursuing full automation.

Figures (5)

• Figure 1: Side-by-side examples of research framings. The fact-checking paper (left, fc-example-new) presents a clear assisted external fact-checking framing with specified users, goals, and methods. The hate speech paper (right, hatespeech-example) has an underspecified framing due to unspecified model users.
• Figure 2: Our system infers a paper's research framings by: (1) estimating epistemic element likelihoods; (2) applying logical rules linking elements to framings; (3) generating explanations summarizing the likelihoods computed in steps 1 and 2 as reasoning for classification; (4) using an LLM to refine predictions using the explanations, context, and in-context examples. Numbers represent log-likelihoods of predicted elements or framings. Examples are from fc-example (top) and hatespeech-example (bottom).
• Figure 3: Example illustrating how the logical rule for Vague Identification is derived from the decision tree.
• Figure 4: Distribution of research framings identified by human annotators in our collected hate speech detection papers.
• Figure 5: Comparison of the distribution of research framings in our analysis (2023 onwards) and that of intendeduses (until 2023).