Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models

TL;DR

It is found that in both base and instruction-tuned LMs, LogProbs offers a more reliable measure of semantic plausibility than direct zero-shot prompting, which yields inconsistent and often poor results.

Abstract

Semantic plausibility (e.g. knowing that "the actor won the award" is more likely than "the actor won the battle") serves as an effective proxy for general world knowledge. Language models (LMs) capture vast amounts of world knowledge by learning distributional patterns in text, accessible via log probabilities (LogProbs) they assign to plausible vs. implausible outputs. The new generation of instruction-tuned LMs can now also provide explicit estimates of plausibility via prompting. Here, we evaluate the effectiveness of LogProbs and basic prompting to measure semantic plausibility, both in single-sentence minimal pairs (Experiment 1) and short context-dependent scenarios (Experiment 2). We find that (i) in both base and instruction-tuned LMs, LogProbs offers a more reliable measure of semantic plausibility than direct zero-shot prompting, which yields inconsistent and often poor results; (ii) instruction-tuning generally does not alter the sensitivity of LogProbs to semantic plausibility (although sometimes decreases it); (iii) across models, context mostly modulates LogProbs in expected ways, as measured by three novel metrics of context-sensitive plausibility and their match to explicit human plausibility judgments. We conclude that, even in the era of prompt-based evaluations, LogProbs constitute a useful metric of semantic plausibility, both in base and instruction-tuned LMs.

Figures (9)

• Figure 1: Results of sentence plausibility judgment performance across models and datasets, using implicit (LogProbs) measures vs. Prompting with the best-performing prompt (Sentence Choice I). Complete prompting results are shown in SI §\ref{['sec:si-complete-prompting']}, Figure \ref{['fig:si-mainresult']}.
• Figure 2: Base vs. instruct model performance in active and passive sentence pairs
• Figure 3: Target word LogProbs replicate patterns of human sentence sensibility judgments. Human data from jouravlev2019tracking. Bars indicate average plausibility of sentences (Human) and average target word log likelihoods (LMs). Dots represent individual sentence scores (averaged across the participant pool for Human).
• Figure 4: Replicating the sensibility-judgment task in LMs using prompting via the adjusted Sentence Judgment prompt in §\ref{['sec:si-jouralev-prompting']}. Human data from jouravlev2019tracking. We use a barplot to visually set apart this prompt-based comparison vs. LogProbs-based ones in Figures \ref{['fig:jouravlev2019']}, \ref{['fig:jouravlev2019-sentence-ll']}.
• Figure 5: Results of implicit vs. explicit plausibility judgment performance experiments