Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complexity as Advantage: A Regret-Based Perspective on Emergent Structure

Oshri Naparstek

TL;DR

This work reframes complexity as the dispersion of predictive regret across a family of observers, rather than an intrinsic property of a source, enabling a relativistic and computable notion of depth. It defines Complexity-as-Advantage (CAA) via L(A;X), L^*(X), R(A;X), and its variance across observer priors, and specializes to log-loss with Markov predictors to connect to predictive information and excess entropy. The authors establish theoretical links to MDL and Kolmogorov complexity, offering a practical interpretation of logical depth as budgeted advantage profiles, and validate the framework with empirical demonstrations on tunable sources, cryptographic ladders, and cellular automata. The results show that CAA reveals exploitable structure invisible to single-observer metrics and provides a versatile diagnostic for learning, evolution, and agent design, with broad implications for dataset difficulty, inductive bias, and intrinsic motivation.

Abstract

We introduce Complexity as Advantage (CAA), a framework that defines the complexity of a system relative to a family of observers. Instead of measuring complexity as an intrinsic property, we evaluate how much predictive regret a system induces for different observers attempting to model it. A system is complex when it is easy for some observers and hard for others, creating an information advantage. We show that this formulation unifies several notions of emergent behavior, including multiscale entropy, predictive information, and observer-dependent structure. The framework suggests that "interesting" systems are those positioned to create differentiated regret across observers, providing a quantitative grounding for why complexity can be functionally valuable. We demonstrate the idea through simple dynamical models and discuss implications for learning, evolution, and artificial agents.

Complexity as Advantage: A Regret-Based Perspective on Emergent Structure

TL;DR

This work reframes complexity as the dispersion of predictive regret across a family of observers, rather than an intrinsic property of a source, enabling a relativistic and computable notion of depth. It defines Complexity-as-Advantage (CAA) via L(A;X), L^*(X), R(A;X), and its variance across observer priors, and specializes to log-loss with Markov predictors to connect to predictive information and excess entropy. The authors establish theoretical links to MDL and Kolmogorov complexity, offering a practical interpretation of logical depth as budgeted advantage profiles, and validate the framework with empirical demonstrations on tunable sources, cryptographic ladders, and cellular automata. The results show that CAA reveals exploitable structure invisible to single-observer metrics and provides a versatile diagnostic for learning, evolution, and agent design, with broad implications for dataset difficulty, inductive bias, and intrinsic motivation.

Abstract

We introduce Complexity as Advantage (CAA), a framework that defines the complexity of a system relative to a family of observers. Instead of measuring complexity as an intrinsic property, we evaluate how much predictive regret a system induces for different observers attempting to model it. A system is complex when it is easy for some observers and hard for others, creating an information advantage. We show that this formulation unifies several notions of emergent behavior, including multiscale entropy, predictive information, and observer-dependent structure. The framework suggests that "interesting" systems are those positioned to create differentiated regret across observers, providing a quantitative grounding for why complexity can be functionally valuable. We demonstrate the idea through simple dynamical models and discuss implications for learning, evolution, and artificial agents.

Paper Structure

This paper contains 47 sections, 3 theorems, 28 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Decision Theory and Regret.
  5. Algorithmic and Statistical Information Theory.
  6. Predictive Information and Excess Entropy.
  7. Complexity in Machine Learning.
  8. Summary.
  9. Complexity-as-Advantage
  10. General Definition
  11. Specialization: Log-Loss and Markov Predictors
  12. Empirical demonstrations
  13. Experiment I: A tunable source and the U-curve
  14. Source.
  15. Observers and loss.
  16. ...and 32 more sections

Key Result

Lemma 3.3

Let $\mathcal{A}=\{A_{\text{naive}},A_{\text{soph}}\}$ with $L(A_{\text{soph}};X)\le L(A_{\text{naive}};X)$. Set $\Delta L \triangleq L(A_{\text{naive}};X)-L(A_{\text{soph}};X)\ge 0$ and let $\pi(A_{\text{naive}})=p$. Then In particular, for the uniform prior $p=\tfrac{1}{2}$,

Figures (7)

  • Figure 1: Gap $\Delta L$ versus $p$ for both observer pairs. Pair A (period-$2$, orders $1$ vs. $3$) is mostly monotone; Pair B (period-$6$, orders $3$ vs. $5$) shows a clear U-shape. Shaded bands: mean$\pm$std over $B$ sequences.
  • Figure 2: $\mathop{\mathrm{CAA}}\nolimits$ versus $p$ (uniform prior); corresponds to Fig. \ref{['fig:u-gap']} via Eq. \ref{['eq:caa-twoalg']}.
  • Figure 3: Relativistic complexity: $\mathop{\mathrm{CAA}}\nolimits_{\max}$ (gap) for four source$\times$observer combinations. Crypto/Search is high because decryption collapses uncertainty; Crypto/Stat is small with long or pseudo-random keys.
  • Figure 4: Crypto ladder. CAA captures a textbook depth barrier: all advantage is locked until one decisive budget threshold.
  • Figure 5: CA ladders ($k=20$). Rule 90 (shallow) gains early, Rule 30 (chaotic) gains diffusely, Rule 110 (deep) defers improvements to large budgets.
  • ...and 2 more figures

Theorems & Definitions (11)

  • Definition 3.1: Asymptotic Average Loss
  • Definition 3.2: Complexity-as-Advantage (CAA)
  • Lemma 3.3: Two-Algorithm Closed Form
  • proof
  • Remark 3.4
  • Definition 3.5: Markov predictors under log-loss
  • Proposition 3.6: Adjacent Markov orders
  • Theorem 3.7: CAA gaps and Excess Entropy
  • proof : Proof sketch
  • Remark 3.8
  • ...and 1 more