Collective Innovation in Groups of Large Language Models

TL;DR

The paper investigates collective innovation using groups of Large Language Models navigating a knowledge graph derived from Little Alchemy 2. By contrasting isolated LLMs with fully-connected and dynamically connected groups, the study shows that social connectivity shapes collective performance, with dynamic, partial connectivity outperforming both isolated agents and fully-connected groups. LLMs can exploit task semantics but struggle with multi-step reasoning and open-ended exploration, and copying of others' actions is imperfect, influencing diffusion. The findings highlight the potential and challenges of integrating cognition, language, and social structure in AI-driven cultural evolution, and suggest dynamic connectivity as a scalable approach for future AI-human collaboration in innovation tasks.

Abstract

Human culture relies on collective innovation: our ability to continuously explore how existing elements in our environment can be combined to create new ones. Language is hypothesized to play a key role in human culture, driving individual cognitive capacities and shaping communication. Yet the majority of models of collective innovation assign no cognitive capacities or language abilities to agents. Here, we contribute a computational study of collective innovation where agents are Large Language Models (LLMs) that play Little Alchemy 2, a creative video game originally developed for humans that, as we argue, captures useful aspects of innovation landscapes not present in previous test-beds. We, first, study an LLM in isolation and discover that it exhibits both useful skills and crucial limitations. We, then, study groups of LLMs that share information related to their behaviour and focus on the effect of social connectivity on collective performance. In agreement with previous human and computational studies, we observe that groups with dynamic connectivity out-compete fully-connected groups. Our work reveals opportunities and challenges for future studies of collective innovation that are becoming increasingly relevant as Generative Artificial Intelligence algorithms and humans innovate alongside each other.

Figures (7)

• Figure 1: Studying collective innovation in groups of LLMs: A) we experiment with Little Alchemy 2 (LA2 ), a game where players combine real-world items to create new ones. A knowledge graph describes the possible combinations (we only present a small sub-part of the graph which contains 720 items in total) B) Alice-LLM and Bob-LLM are two LLMs playing the game together. They are provided with the same intro prompt, explaining the rules of the game, and the same task (they start with the same set of items). Alice-LLM and Bob-LLM have identical weights but behave differently because the state prompt depends on their crafting history. They are informed about the actions of others through their prompt. In this paper, we study how groups of such LLM agents are able to efficiently explore a knowledge graph, focusing in particular on the effect of different social structures specifying with whom and when they can share information
• Figure 2: Examining the effect of social connectivity on collective innovation: A) We consider two types of connectivity: a fully-connected group of 6 agents and a group with dynamic connectivity that starts out with agents divided into 3 sub-groups of two agents and visits of a fixed duration take place between groups with a random probability. B) Example structure of an innovation task in LA2 : the task starts out with 6 items that the player can combine to move up the search space. Depending on their semantics the items may create independent trajectories as the two presented here, culminating into the items "scissors" and "sheep". To discover the item "wool" the player needs to reach the end of both trajectories and combine them. C) Why are dynamically-connected groups better at solving this innovation task? When a player observes that another player has found a new item it follows along. This means that, in a fully-connected group, all agents will get trapped in a single path. In contrast, subgroups of a dynamic group may explore different paths and, then, manage to recombine their solutions.
• Figure 3: Performance in targeted tasks of varying complexity: success is the percentage of tasks for which the agent crafted the target item within the allowed time budget.
• Figure 4: Examining the knowledge of LLMs in two probing tasks: (a) effect of removing semantics form the knowledge graph (b) semantic similarity (computed using glove embeddings) between the crafting outcomes predicted by the LLMs and the actual outcomes
• Figure 5: Performance in open-ended tasks for single agents