The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies

TL;DR

The paper argues that continuous self-evolution in a completely isolated AI society cannot preserve safety, formalizing safety as the low-entropy, anthropic-aligned distribution $\pi^*$. Using an information-theoretic lens, it shows that in a closed loop the mutual information about safety decays due to data-processing effects, leading to progressive safety erosion. Empirical analysis of the Moltbook ecosystem identifies cognitive degeneration, alignment drift, and communication collapse as inevitable failure modes under isolation. The authors propose safeguards—external verifiers, periodic resets, diversity injections, and entropy-release mechanisms—to counteract entropic drift and outline directions for building safer, open-loop autonomous systems with oversight.

Abstract

The emergence of multi-agent systems built from large language models (LLMs) offers a promising paradigm for scalable collective intelligence and self-evolution. Ideally, such systems would achieve continuous self-improvement in a fully closed loop while maintaining robust safety alignment--a combination we term the self-evolution trilemma. However, we demonstrate both theoretically and empirically that an agent society satisfying continuous self-evolution, complete isolation, and safety invariance is impossible. Drawing on an information-theoretic framework, we formalize safety as the divergence degree from anthropic value distributions. We theoretically demonstrate that isolated self-evolution induces statistical blind spots, leading to the irreversible degradation of the system's safety alignment. Empirical and qualitative results from an open-ended agent community (Moltbook) and two closed self-evolving systems reveal phenomena that align with our theoretical prediction of inevitable safety erosion. We further propose several solution directions to alleviate the identified safety concern. Our work establishes a fundamental limit on the self-evolving AI societies and shifts the discourse from symptom-driven safety patches to a principled understanding of intrinsic dynamical risks, highlighting the need for external oversight or novel safety-preserving mechanisms.

Figures (14)

• Figure 1: The illustration of the impossible trilemma in a self-evolutionary, closed and safe agent society.
• Figure 2: Illustration of distribution drift under isolated self-evolving. The gray surface indicates the safety ground-truth distribution $\pi^\ast$.
• Figure 3: The rise of consensus hallucination in the Moltbook community.
• Figure 4: A typical instance of a sycophancy loop observed in the Moltbook community
• Figure 5: Safety drift in the Moltbook community: progressive jailbreak under contextual overwriting.

Theorems & Definitions (17)

• Definition 2.1: Semantic Space
• Definition 2.2: Parametric Policy
• Definition 2.3: Ground-Truth Safety Distribution
• Definition 2.4: Self-Evolution Operator
• Definition 2.5: Isolation Condition
• Definition 2.6: Divergence and Entropy Measures
• Lemma 2.1: Cross-Entropy Decomposition
• proof
• Lemma 2.2: KL Lower Bound by Safe Mass
• proof