LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy

Abstract

Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.

Figures (8)

• Figure 1: ROC curves for three models on TriviaQA across seven temperatures (T = 0.1--2.0). Warmer colours indicate higher temperatures. Curves shift outward with increasing temperature, indicating increasing AUC.
• Figure 2: Temperature effects on SDT parameters (TriviaQA). Left: AUC increases monotonically ($\rho$$\geq$ 0.964). Centre: $d_a$ trend. Right: Criterion c shifts monotonically from liberal toward less liberal. Shaded bands: bootstrap 95% CIs.
• Figure 3: SDT operating points at T = 1.0. Position: ($d_a$, c); fill: ECE. Edge colour: model identity. Circles: TriviaQA; squares: NQ. Models occupy distinct positions invisible to ECE alone.
• Figure 4: Bland-Altman plot: $d_a$ (Paradigm A, T = 1.0) versus $d'_{\text{4AFC}}$ (Paradigm B) across domains. Systematic negative bias reflects near-ceiling 4AFC inflating $d'_{\text{4AFC}}$.
• Figure 5: z-ROC plots (TriviaQA, all temperatures overlaid). Black line: T = 1.0 regression. Slopes < 1.0 confirm unequal variance. Instruct models: 0.52--0.63; base model: 0.78.