Potential and Perils of Large Language Models as Judges of Unstructured Textual Data

TL;DR

This paper investigates using large language models as evaluative judges to assess the thematic alignment of summaries produced by other LLMs, comparing their judgments to human evaluations via Cohen's kappa, Spearman's rho, and Krippendorff's alpha. It employs Anthropic Claude as the initial evaluator and multiple models (Titan Express, Nova Pro, Llama) as judges to gauge inter-model agreement, using a real-world, open-text survey dataset of over 13,000 responses segmented into 70 groups. The results show that LLM-as-judge approaches achieve moderate agreement with humans and strong inter-model consistency, but humans remain better at detecting nuanced or context-specific misalignments; the study emphasizes biases and the need for robust, multi-metric evaluation frameworks. The work contributes to scalable AI-assisted text analysis while highlighting ethical considerations, biases, and the necessity for careful prompt design and human oversight in high-stakes organizational settings.

Abstract

Rapid advancements in large language models have unlocked remarkable capabilities when it comes to processing and summarizing unstructured text data. This has implications for the analysis of rich, open-ended datasets, such as survey responses, where LLMs hold the promise of efficiently distilling key themes and sentiments. However, as organizations increasingly turn to these powerful AI systems to make sense of textual feedback, a critical question arises, can we trust LLMs to accurately represent the perspectives contained within these text based datasets? While LLMs excel at generating human-like summaries, there is a risk that their outputs may inadvertently diverge from the true substance of the original responses. Discrepancies between the LLM-generated outputs and the actual themes present in the data could lead to flawed decision-making, with far-reaching consequences for organizations. This research investigates the effectiveness of LLM-as-judge models to evaluate the thematic alignment of summaries generated by other LLMs. We utilized an Anthropic Claude model to generate thematic summaries from open-ended survey responses, with Amazon's Titan Express, Nova Pro, and Meta's Llama serving as judges. This LLM-as-judge approach was compared to human evaluations using Cohen's kappa, Spearman's rho, and Krippendorff's alpha, validating a scalable alternative to traditional human centric evaluation methods. Our findings reveal that while LLM-as-judge offer a scalable solution comparable to human raters, humans may still excel at detecting subtle, context-specific nuances. Our research contributes to the growing body of knowledge on AI assisted text analysis. Further, we provide recommendations for future research, emphasizing the need for careful consideration when generalizing LLM-as-judge models across various contexts and use cases.