Quo Vadis, HCOMP? A Review of 12 Years of Research at the Frontier of Human Computation and Crowdsourcing

TL;DR

The paper assesses 12 years of research at the Conference on Human Computation and Crowdsourcing (HCOMP) through Kuhn's paradigm-shifts lens to determine whether generative AI drives a fundamental paradigm shift or a gradual topic reorientation. Using a multi-method approach (text embeddings, clustering, co-word analysis, and cross-conference comparisons), the authors map topic evolution from classic crowdsourcing themes to human–AI collaboration, interpretability, and ethics. Their findings indicate a gradual gestalt-like shift beginning around 2018, with increasing overlap with venues like FAccT, IUI, and UMAP, and a persistent—but not yet revolutionary—redefinition of HCOMP's epistemic foundations. The study argues that while HCOMP is evolving due to AI disruptions, it has not yet undergone a Kuhnian revolution, instead broadening its scope and potentially benefiting from closer ties with Collective Intelligence (CI); the work provides a data-driven basis for strategic decisions about the conference's future and research directions.

Abstract

The field of human computation and crowdsourcing has historically studied how tasks can be outsourced to humans. However, many tasks previously distributed to human crowds can today be completed by generative AI with human-level abilities, and concerns about crowdworkers increasingly using language models to complete tasks are surfacing. These developments undermine core premises of the field. In this paper, we examine the evolution of the Conference on Human Computation and Crowdsourcing (HCOMP) - a representative example of the field as one of its key venues - through the lens of Kuhn's paradigm shifts. We review 12 years of research at HCOMP, mapping the evolution of HCOMP's research topics and identifying significant shifts over time. Reflecting on the findings through the lens of Kuhn's paradigm shifts, we suggest that these shifts do not constitute a paradigm shift. Ultimately, our analysis of gradual topic shifts over time, combined with data on the evident overlap with related venues, contributes a data-driven perspective to the broader discussion about the future of HCOMP and the field as a whole.

Figures (8)

• Figure 1: Co-word occurrences at the HCOMP Conference over time (order-insensitive, considering only co-words that appear in more than one year)
• Figure 2: Comparison of article titles in HCOMP, ACM Collective Intelligence (CI) and ACM Conference on Fairness, Accountability, and Transparency (FAccT). Titles of more recent articles are more opaque.
• Figure 3: Cosine distances between centroids of article title embeddings in subsequent conference years at the HCOMP Conference (red) and six related conferences. One can note that older venues, such as CSCW and AAMAS, display little centroid movement. This is also evident at ACM FAccT.
• Figure 4: Shared keywords between the HCOMP conference and six related conferences over time (based on article titles)
• Figure 5: Shared keywords in the article titles at HCOMP (left) and six related conferences (ACM CI, ACM FAccT, ACM CSCW, ACM IUI, ACM UMAP, and AAMAS)