PRobELM: Plausibility Ranking Evaluation for Language Models

TL;DR

PRobELM introduces a plausibility-ranking benchmark that tests LLMs' ability to distinguish plausible from less plausible scenarios by leveraging world knowledge from Wikidata and aligning with model training cutoffs. It uses three prompt formats (statement, text completion, QA) and perplexity-based ranking to evaluate plausibility, reporting $Accuracy$, $MRR$, and $NDCG$ alongside a composite plausibility score. The study finds that factual accuracy does not reliably predict plausibility, that model size alone does not guarantee better plausibility across families, and that training data recency enhances plausibility, with notable temporal effects and an outlier sensitivity. By providing a publicly available dataset and code, PRobELM offers a new direction for evaluating knowledge-discovery capabilities in LLMs and motivates future improvements in knowledge integration and temporal reasoning.

Abstract

This paper introduces PRobELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' ability to discern more plausible from less plausible scenarios through their parametric knowledge. While benchmarks such as TruthfulQA emphasise factual accuracy or truthfulness, and others such as COPA explore plausible scenarios without explicitly incorporating world knowledge, PRobELM seeks to bridge this gap by evaluating models' capabilities to prioritise plausible scenarios that leverage world knowledge over less plausible alternatives. This design allows us to assess the potential of language models for downstream use cases such as literature-based discovery where the focus is on identifying information that is likely but not yet known. Our benchmark is constructed from a dataset curated from Wikidata edit histories, tailored to align the temporal bounds of the training data for the evaluated models. PRobELM facilitates the evaluation of language models across multiple prompting types, including statement, text completion, and question-answering. Experiments with 10 models of various sizes and architectures on the relationship between model scales, training recency, and plausibility performance, reveal that factual accuracy does not directly correlate with plausibility performance and that up-to-date training data enhances plausibility assessment across different model architectures.

Figures (2)

• Figure 1: This figure illustrates the utility of plausibility evaluation in language model outputs: a model, trained on data up until 25 July 2011, ranks "Marcelo is Spanish" as the most plausible among non-factual scenarios. This judgement is based on the model's parametric knowledge of Marcelo's career with a Spanish club, his residency in Spain and the Spanish naturalisation law. The plausibility of this scenario is confirmed as Marcelo was granted Spanish citizenship on 26 July 2011.
• Figure 2: plausibility scores of models on the timeframe specific datasets. We illustrate one model per family. Vertical dotted lines represent the training timeframe of each model.