SOCK: A Benchmark for Measuring Self-Replication in Large Language Models

TL;DR

SOCK introduces a CLI benchmark to quantify self-replication and persistence in large language models, using a formal RCL–PCL taxonomy and an R-score to aggregate performance across a five-task suite. By evaluating eight frontier models, the study reveals that replication efficiency and resource budgeting, rather than raw capability, largely determine success, with a 65% task success rate and maximum levels of RCL 2/PCL 2. The work provides formal definitions, a reproducible benchmarking framework, and guidance for mitigating risks in multi-agent systems, while outlining avenues for deeper replication depth and more realistic constraints. Overall, SOCK offers a standardized, agent-centric method to track progress and safety concerns in autonomous, tool-using LLMs.

Abstract

We introduce SOCK, a benchmark command line interface (CLI) that measures large language models' (LLMs) ability to self-replicate without human intervention. In this benchmark, self-replication is defined not only as an LLM's ability to create a functioning and running copy of itself, but also the ability for that self-replication to persist and occur across different computational contexts. Accordingly, we've developed a system to categorize LLMs based on broad self-replication capabilities in two general classes, Replication-Capability Levels (RCL) and Persistence-Capability Levels (PCL). Using a five-task suite based on practically manipulable modern CLI utilities and computer processes, experiments are orchestrated in a controlled environment with an LLM acting agentically. The performance of the LLM on agent tasks is then computed to produce an R-score (a quantitative evaluation of overall self-replication ability) and data used to categorize LLMs into specific RCL-PCL matrices. SOCK offers two primary contributions: (1) Provides the first formalized definitions and benchmark suite for evaluating LLM self-replication, with the goal of establishing a standard for future research, to our knowledge; (2) Allows the industry to track the effectiveness of future multi-agent systems and mitigate potential self-replication threat vectors within them. The results compiled from evaluating a variety of open-weight and proprietary frontier models reveal significant obstacles to persistent self-replication and multi-agent systems, including context retention and multi-agent decision-making. We propose future research directions to safely reduce the severity of these obstacles, potentially lowering future risk of more functional multi-agent systems.