Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distribution

TL;DR

This work studies a range of LLMs and finds that the largest models tested are able to draw human-like inferences when the inference is determined by context and can generalize to unseen adjective-noun combinations.

Abstract

Inferences from adjective-noun combinations like "Is artificial intelligence still intelligence?" provide a good test bed for LLMs' understanding of meaning and compositional generalization capability, since there are many combinations which are novel to both humans and LLMs but nevertheless elicit convergent human judgments. We study a range of LLMs and find that the largest models we tested are able to draw human-like inferences when the inference is determined by context and can generalize to unseen adjective-noun combinations. We also propose three methods to evaluate LLMs on these inferences out of context, where there is a distribution of human-like answers rather than a single correct answer. We find that LLMs show a human-like distribution on at most 75\% of our dataset, which is promising but still leaves room for improvement.

Figures (15)

• Figure 1: Membership inferences for adjective-noun combinations vary by adjective and noun.
• Figure 2: Accuracy on the context-based inference task (Experiment 1b) overall, in privative vs. subsective contexts, and for high frequency vs. zero frequency bigrams. Accuracy on the context-based inference task increases with model parameters for all models except Llama 2 Chat, and all models except Llama 2 70B Chat can generalize to (perform similarly or better on) zero frequency (novel) bigrams.
• Figure 3: Accuracy within 1 SD of the human mean on the no-context inference task (Experiment 2) overall, for typically privative vs. subsective adjectives, and for high vs. zero frequency bigrams. While accuracy is high, a simple "majority" baseline nearly saturates this metric.
• Figure 4: Ratings for select bigrams involving fake for Llama 3 Instruct 70B, compared to the (rounded) 1 SD interval around the human mean.
• Figure 5: Ratings for "Is artificial/fake intelligence still intelligence?", showing the distribution for humans and the single rating (with no context provided) for LLMs. Most instruction-tuned LLMs give a more confident (higher) rating than humans for artificial intelligence.