How Do Hackathons Foster Creativity? Towards AI Collaborative Evaluation of Creativity at Scale

TL;DR

This paper tackles the lack of large-scale quantitative analysis of creativity in hackathons by operationalizing creativity as novelty and usefulness and applying this framework to a Devpost-derived dataset of 193,353 projects reduced to 10,363 for analysis. It employs a five-C creativity lens (Creators, Collaborations, Contexts, Creations, Consumption) and uses a Monte Carlo-based novelty metric alongside winner-tag driven usefulness to identify 619 creative projects, subsequently modeling the predictors of creativity with a mixed random effects logistic regression that achieves a near-perfect fit ($R^2_m ≈ 0.967$, $R^2_c ≈ 0.970$). In parallel, it investigates Large Language Models as-a-judge to augment large-scale creativity evaluation, comparing their ratings to human judgments and demonstrating notable variability across models, while advocating a human-AI collaborative approach to enhance scalability and insight. The work provides practical take-aways for organizers and outlines opportunities and challenges in evaluating creativity at scale, including methodological refinements, temporal/contextual considerations, and environmental impacts of AI-assisted research.

Abstract

Hackathons have become popular collaborative events for accelerating the development of creative ideas and prototypes. There are several case studies showcasing creative outcomes across domains such as industry, education, and research. However, there are no large-scale studies on creativity in hackathons which can advance theory on how hackathon formats lead to creative outcomes. We conducted a computational analysis of 193,353 hackathon projects. By operationalizing creativity through usefulness and novelty, we refined our dataset to 10,363 projects, allowing us to analyze how participant characteristics, collaboration patterns, and hackathon setups influence the development of creative projects. The contribution of our paper is twofold: We identified means for organizers to foster creativity in hackathons. We also explore the use of large language models (LLMs) to augment the evaluation of creative outcomes and discuss challenges and opportunities of doing this, which has implications for creativity research at large.

Figures (9)

• Figure 1: Number of projects remaining after each pre-processing step (left). This process is described in detail in section \ref{['Data collection and pre-processing']}; Extracted variables for analysis (mid). The analysis is described in detail in section \ref{['novelty']}; Resulting prediction of creative projects (right). The analysis results are described in detail in section \ref{['predicting creative projects']}.
• Figure 2: Variables in our dataset. The underlined Variables are processed from the collected data.
• Figure 3: Top: The percentage of Creative Projects of the number of Teams by the number of Team Members. Bottom: The distribution of creative and non-creative projects by the number of participants in hackathons (outliers removed outside the 75% percentile).
• Figure 4: Score distribution of Novelty and Usefulness scores of LLM-as-a-judge on the subset of hackathons.
• Figure 5: Heatmaps of inter-rater reliability for both novelty and usefulness in hackathons in different orders of granularity, i.e., how the scores are bucketed. (a-b) contain scores (1-2-3, 4-5), (c-d) contains scores (1-2, 3, 4-5), and (e-f) contains (1-5) in separate buckets.