Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-as-evaluator in Strategy Research: A Normative, Variance-Aware Protocol

Arnaldo Camuffo, Alfonso Gambardella, Saeid Kazemi, Jakub Malachowski, Abhinav Pandey

TL;DR

The paper investigates the reliability of using large language models as evaluators in strategy research, revealing substantial instability in outputs due to prompts, context, sampling, extraction, and model differences. It develops a normative, variance-aware protocol to convert LLM outputs into auditable measurements, with preregistration and transparent reporting to improve validity. Through controlled experiments, it demonstrates how output level, prompt design, and model choice drive sizeable swings in evaluation results, including cross-model disagreements. The work then offers concrete guidelines, diagnostics, and an implementation blueprint to build measurement infrastructure that makes LLM-based evaluations rigorous, interpretable, and replicable for scholarly research.

Abstract

Large language models (LLMs) are becoming essential tools for strategy scholars who need to evaluate text corpora at scale. This paper provides a systematic analysis of the reliability of LLM-as-evaluator in strategy research. After classifying the typical ways in which LLMs can be deployed for evaluation purposes in strategy research, we draw on the specialised AI literature to analyse their properties as measurement instruments. Our empirical analysis reveals substantial instability in LLMs' evaluation output, stemming from multiple factors: the specific phrasing of prompts, the context provided, sampling procedures, extraction methods, and disagreements across different models. We quantify these effects and demonstrate how this unreliability can compromise the validity of research inferences drawn from LLM-generated evaluations. To address these challenges, we develop a comprehensive protocol that is variance-aware, normative, and auditable. We provide practical guidance for flexible implementation of this protocol, including approaches to preregistration and transparent reporting. By establishing these methodological standards, we aim to elevate LLM-based evaluation of business text corpora from its current ad hoc status to a rigorous, actionable, and auditable measurement approach suitable for scholarly research.

LLM-as-evaluator in Strategy Research: A Normative, Variance-Aware Protocol

TL;DR

The paper investigates the reliability of using large language models as evaluators in strategy research, revealing substantial instability in outputs due to prompts, context, sampling, extraction, and model differences. It develops a normative, variance-aware protocol to convert LLM outputs into auditable measurements, with preregistration and transparent reporting to improve validity. Through controlled experiments, it demonstrates how output level, prompt design, and model choice drive sizeable swings in evaluation results, including cross-model disagreements. The work then offers concrete guidelines, diagnostics, and an implementation blueprint to build measurement infrastructure that makes LLM-based evaluations rigorous, interpretable, and replicable for scholarly research.

Abstract

Large language models (LLMs) are becoming essential tools for strategy scholars who need to evaluate text corpora at scale. This paper provides a systematic analysis of the reliability of LLM-as-evaluator in strategy research. After classifying the typical ways in which LLMs can be deployed for evaluation purposes in strategy research, we draw on the specialised AI literature to analyse their properties as measurement instruments. Our empirical analysis reveals substantial instability in LLMs' evaluation output, stemming from multiple factors: the specific phrasing of prompts, the context provided, sampling procedures, extraction methods, and disagreements across different models. We quantify these effects and demonstrate how this unreliability can compromise the validity of research inferences drawn from LLM-generated evaluations. To address these challenges, we develop a comprehensive protocol that is variance-aware, normative, and auditable. We provide practical guidance for flexible implementation of this protocol, including approaches to preregistration and transparent reporting. By establishing these methodological standards, we aim to elevate LLM-based evaluation of business text corpora from its current ad hoc status to a rigorous, actionable, and auditable measurement approach suitable for scholarly research.
Paper Structure (99 sections, 11 equations, 4 figures, 11 tables)

This paper contains 99 sections, 11 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Problem Definition and Background
  3. Why a protocol is needed
  4. Background research on LLM-as-evaluator
  5. When not to use LLM evaluation
  6. Five sources of variance
  7. LLM-as-evaluator: Mechanism and Volatility
  8. What LLM-as-evaluator does (model mechanism)
  9. Inputs and sequence construction.
  10. Embedding and positional encoding.
  11. Transformer computation.
  12. Logits, softmax, and decoding.
  13. Putting the pieces together.
  14. Specificity of LLMs’ application to evaluation tasks
  15. Level 1: single-label (one question, one analysable answer)
  16. ...and 84 more sections

Figures (4)

  • Figure 1: The five sources of variance generation for LLM-as-evaluator output, with associated mechanisms. Arrows indicate how each source contributes variance to LLM evaluation output; diagnostics and controls operate within each box.
  • Figure 2: Level 1 format sensitivity: distributions for binary, point-wise, and probability encodings by model ($n{=}20$ per encoding).
  • Figure 3: Level 1 A/B swap: aligned original vs. swap distributions by model. Overlap implies order invariance; divergence implies position effects ($n{=}20$ per condition).
  • Figure 4: Score distributions under low vs. high reasoning (GPT-5 reasoning).