Who is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition
Sarina Xi, Orelia Pi, Miaomiao Zhang, Becca Xiong, Jacqueline Ng Lane, Nihar B. Shah
TL;DR
This paper tackles judge-appointment in high-stakes startup competitions by developing HLSE, a Hybrid Lexical-Semantic Similarity Ensemble that fuses TF-IDF, transformer embeddings, and hybrid representations to score judge-venture similarity. It couples HLSE with the PeerReview4All assignment algorithm to optimize fairness and expertise coverage, and evaluates performance against human expert assignments in a real deployment. In a head-to-head, blinded study with 309 judge-venture pairs, algorithmic matches achieved human-expert-level quality (mean scores: manual $3.94$ vs algorithmic $3.90$; $AUC=0.48$, $p=0.40$) while dramatically reducing assignment time from about a week to hours. The results demonstrate AI-assisted judge assignment can preserve quality at scale, offering practical pathways for deploying scalable, high-quality evaluations in high-stakes contexts and guiding future work on robustness and cross-domain applications.
Abstract
There is growing interest in applying artificial intelligence (AI) to automate and support complex decision-making tasks. However, it remains unclear how algorithms compare to human judgment in contexts requiring semantic understanding and domain expertise. We examine this in the context of the judge assignment problem, matching submissions to suitably qualified judges. Specifically, we tackled this problem at the Harvard President's Innovation Challenge, the university's premier venture competition awarding over \$500,000 to student and alumni startups. This represents a real-world environment where high-quality judge assignment is essential. We developed an AI-based judge-assignment algorithm, Hybrid Lexical-Semantic Similarity Ensemble (HLSE), and deployed it at the competition. We then evaluated its performance against human expert assignments using blinded match-quality scores from judges on $309$ judge-venture pairs. Using a Mann-Whitney U statistic based test, we found no statistically significant difference in assignment quality between the two approaches ($AUC=0.48, p=0.40$); on average, algorithmic matches are rated $3.90$ and manual matches $3.94$ on a 5-point scale, where 5 indicates an excellent match. Furthermore, manual assignments that previously required a full week could be automated in several hours by the algorithm during deployment. These results demonstrate that HLSE achieves human-expert-level matching quality while offering greater scalability and efficiency, underscoring the potential of AI-driven solutions to support and enhance human decision-making for judge assignment in high-stakes settings.