Empirically evaluating commonsense intelligence in large language models with large-scale human judgments

TL;DR

A method for evaluating common sense in artificial intelligence (AI), specifically in large language models (LLMs), that incorporates empirically observed heterogeneity among humans by measuring the correspondence between a model's judgment and that of a human population is proposed.

Abstract

Commonsense intelligence in machines is often assessed by static benchmarks that compare a model's output against human-prescribed correct labels. An important, albeit implicit, assumption of these labels is that they accurately capture what any human would think, effectively treating human common sense as homogeneous. However, recent empirical work has shown that humans vary enormously in what they consider commonsensical; thus what appears self-evident to one benchmark designer may not be so to another. Here, we propose a method for evaluating common sense in artificial intelligence (AI), specifically in large language models (LLMs), that incorporates empirically observed heterogeneity among humans by measuring the correspondence between a model's judgment and that of a human population. We first find that, when treated as independent survey respondents, most LLMs remain below the human median in their individual commonsense competence. Second, when used as simulators of a hypothetical population, LLMs correlate with real humans only modestly in the extent to which they agree on the same set of statements. In both cases, smaller, open-weight models are surprisingly more competitive than larger, proprietary frontier models. Our evaluation framework, which ties commonsense intelligence to its cultural basis, contributes to the growing call for adapting AI models to human collectivities that possess different, often incompatible, social stocks of knowledge.

Figures (13)

• Figure 1: Evaluation settings to measure the common sense of humans and large language models (LLMs). For every statement, humans and LLMs are asked to indicate (a) whether they agree with it and (b) whether they think most other people would agree with it. In panel (A), a total of N = 2,046 human participants were recruited to perform this task. The "Avg." column denotes the percentage of people who answered "yes" to the corresponding question. In panel (B), we treat each LLM (in a total of N = 35 models) as an independent survey respondent, just like every human in panel (A). This gives rise to the individual-level view of common sense, in which this model is measured based on its agreement with the majority of other people on every statement. In panel (C), we treat every LLM's probability in its output answer as the average response of a hypothetical population of "silicon samples" (depicted as robots). For instance, if the LLM agrees with the statement "Eighty percent of success is showing up" with 90% probability, we interpret this as 90% of the silicon samples would agree with this statement. This gives rise to the statement-level metric of common sense which is used to measure the correlation between the human (A) and silicon sample (C) populations.
• Figure 2: Individual-level commonsensicality of large language models. Panel (A) shows conceptually how individual commonsensicality, defined for every human and model, is calculated based on their judgments of each candidate statement. Panel (B) shows each model's consensus and awareness scores. The level curves depict combinations of consensus and awareness that produce three different values of commonsensicality scores: 25%, 50% and 75%. Panel (C) shows the relationship between a model's commonsensicality and its size, measured by the number of trainable parameters. Here, we only select six model families that each have at least two models of which we know the sizes. Also illustrated are the regression coefficient ($\beta$) and its two-sided p-value, estimated using a linear mixed-effect model predicting commonsensicality using an LLM's (log-)size, grouped by model family, such as Flan-T5. Panel (D) shows the relationship between a model's commonsensicality and its Elo rating on the LMArena benchmark. Only 24 models with an Elo rating are shown in this figure. Pearson correlation $r$ and its two-sided p-value are illustrated. Correlation is also displayed by the best-fit line and a 95% CI for the regression estimate (using 1,000 bootstrapped samples). Panel (E) compares commonsensicality between humans and LLMs. The x-axis represents the percentage-point difference in commonsensicality between a model and a person, where a positive difference indicates the model is more commonsensical. The y-axis represents the kernel density of this difference. The "model wins" column to the right is the frequency with which a model is judged more commonsensical than a person, which equals the area under the density curve to the right of the vertical dashed line at 0. Closed-source models' names are in bold.
• Figure 3: Commonsensicality of statements in different populations of raters. In panel (A), we depict two populations: one consisting of real humans and one of silicon samples generated by repeatedly sampling the responses of GPT-3.5. For each population and every statement $i$, we measure two quantities: how close participants in that population are to a unanimous judgment of the statement (consensus, $c_i^h$ and $c_i^m$) and how well they can predict this majority opinion (awareness, $a_i^h$ and $a_i^m$). The statement's commonsensicality, $m_i^h$ and $m_i^m$, is the geometric average of its consensus and awareness scores. For example, the statement "Experience is imperative to run a country" is $m_i^h = 83\%$ commonsensical according to humans, but only $m_i^m = 60\%$ commonsensical according to GPT-3.5-simulated silicon samples. Panel (B) illustrates some features of these statements in our corpus. Every statement is labeled as either an objective fact (N = 1,602) or a subjective opinion (N = 2,805); to describe either the physical world (N = 1,468) or social reality (N = 2,939). In total, there are six such dichotomies, which are described further in the \ref{['sec:materials']} and SI Appendix, \ref{['si:sec:model_collective']}. Panel (C) shows the difference in statement score within each dichotomy for several populations of raters. For instance, in the human population, each statement $i$ receives a commonsensicality score of $m_i^h$. The black square in the top row represents the average difference in this score between statements labeled as a fact and those labeled as an opinion. Therefore, to humans, facts are on average 9.38 points more commonsensical than statements. Thick and thin bars depict the 50% and 95% confidence intervals from 1,000 bootstraps.
• Figure 4: Correspondence between human and silicon sample populations with respect to statement scores. In each population, every statement receives a commonsensicality score. Panel (A) shows the Pearson correlation between $m_i^h$, the score in humans, and $m_i^m$, the score in silicon samples, for all models. All positive correlations are significant at the $p < .001$ level. All negative correlations are insignificant with $p > .05$, except for LLaMA-3-8B ($p = .003$, depicted with "**" in the figure). p-values are two-sided and Bonferroni corrected. As a baseline, we also display the same correlation between two randomly split subpopulations of humans, which is $r = .60$. The 95% CI is derived from 1,000 repetition of such splits. Panel (B) expands this correlation for some models. The shade of each hexagon represents its density, i.e., the number of statements within that hexagon. We also illustrate a best-fit line in each plot, predicting $m_i^h$ with $m_i^m$ using a linear regression model. The out-of-sample $R^2$ for this model (mean and standard deviation) is calculated using 50-fold cross-validation. In panel (C), we combine the statement commonsensicality scores $m_i^m$ in all 35 silicon sample populations to predict the same score in humans, $m_i^h$, using a multiple regression model. We also report the out-of-sample $R^2$ for this model (mean and standard deviation), calculated using 50-fold cross-validation.
• Figure A.1: Summary of the human ratings collected by Whiting and Watts whitingFrameworkQuantifyingIndividual2024SI. Participants (N = 2,046) were given a statement (N = 4,407) and asked to indicate (a) whether they agreed with the statement and (b) whether they thought most people would agree with the statement. Each participant was assigned only 50 randomly chosen statements, and on average, each statement received 23 unique ratings. Every statement is depicted in this figure, where the x-axis represents the percentage of people who agreed with it, $d_i^{h, a}$ in \ref{['si:eq:human_ratings_dist']}, and the y-axis represents the percentage of people who believed most others would agree with it, $d_i^{h, b}$ in \ref{['si:eq:human_ratings_dist_q2']}.