The Limits of Goal-Setting Theory in LLM-Driven Assessment
Mrityunjay Kumar
TL;DR
This study investigates whether increased goal specificity in prompts (as a proxy for measuring goal-setting theory) improves the consistency and reduces the variance of LLM-driven assessment. Using ChatGPT to evaluate 29 student essays across four prompt types (varying presence of rubrics and model solutions), the authors measure intra-run agreement with weighted Cohen's Kappa and analyze across six repetitions per type. The results show that higher specificity does not reliably improve consistency, and variance remains largely unchanged, challenging the notion that LLM evaluators behave like humans under goal-setting assumptions. The findings highlight robustness and input-integration challenges in current LLMs and call for architectural or training-level solutions to achieve reliable, high-stakes educational evaluation.
Abstract
Many users interact with AI tools like ChatGPT using a mental model that treats the system as human-like, which we call Model H. According to goal-setting theory, increased specificity in goals should reduce performance variance. If Model H holds, then prompting a chatbot with more detailed instructions should lead to more consistent evaluation behavior. This paper tests that assumption through a controlled experiment in which ChatGPT evaluated 29 student submissions using four prompts with increasing specificity. We measured consistency using intra-rater reliability (Cohen's Kappa) across repeated runs. Contrary to expectations, performance did not improve consistently with increased prompt specificity, and performance variance remained largely unchanged. These findings challenge the assumption that LLMs behave like human evaluators and highlight the need for greater robustness and improved input integration in future model development.