Minimization of Boolean Complexity in In-Context Concept Learning

TL;DR

The paper investigates whether large language models (LLMs) can learn new concepts from in-context examples and whether they exhibit a human-like simplicity bias. It defines concept complexity via a minimal description length in a formal logical grammar and evaluates two LLM families across complexity classes using carefully balanced prompts. Results show a robust, negative relationship between Boolean complexity and learning accuracy, with Pearson correlations ranging from $0.854$ to $0.961$ (p-values from $0.009$ to $0.065$), indicating that simpler concepts are learned more easily. The findings imply that in-context learning in LLMs shares a bias toward simpler representations with human concept learning, suggesting avenues for cross-disciplinary study and more robust prompt-based concept acquisition in AI systems.

Abstract

What factors contribute to the relative success and corresponding difficulties of in-context learning for Large Language Models (LLMs)? Drawing on insights from the literature on human concept learning, we test LLMs on carefully designed concept learning tasks, and show that task performance highly correlates with the Boolean complexity of the concept. This suggests that in-context learning exhibits a learning bias for simplicity in a way similar to humans.

Figures (4)

• Figure 1: The influence of concept complexity on LLM accuracy at concept learning (see text for how complexity is operationalized). On average, LLM accuracy drops as complexity increases.
• Figure 2: Complexity vs. average accuracy.
• Figure :
• Figure :