Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Complexity an Illusion?

Michael Timothy Bennett

TL;DR

This work interrogates whether complexity causally drives generalisation or is merely an artefact of abstraction. Using a pancomputational, abstraction‑layer formalism with a finite vocabulary, it shows that without abstraction all behaviours have equivalent apparent complexity, making complexity seem illusory; when vocabularies are finite, weakness can masquerade as simplicity and thus correlate with sample efficiency. The authors argue that abstraction is goal-directed, biasing weaker policies to simpler forms under constraints, which explains observed correlations between simplicity and generalisation. The results clarify when complexity-based predictions are meaningful for interactive learning and suggest that observed gains stem from interpretation and constraint structure rather than a fundamental link between complexity and intelligence.

Abstract

Simplicity is held by many to be the key to general intelligence. Simpler models tend to "generalise", identifying the cause or generator of data with greater sample efficiency. The implications of the correlation between simplicity and generalisation extend far beyond computer science, addressing questions of physics and even biology. Yet simplicity is a property of form, while generalisation is of function. In interactive settings, any correlation between the two depends on interpretation. In theory there could be no correlation and yet in practice, there is. Previous theoretical work showed generalisation to be a consequence of "weak" constraints implied by function, not form. Experiments demonstrated choosing weak constraints over simple forms yielded a 110-500% improvement in generalisation rate. Here we show that all constraints can take equally simple forms, regardless of weakness. However if forms are spatially extended, then function is represented using a finite subset of forms. If function is represented using a finite subset of forms, then we can force a correlation between simplicity and generalisation by making weak constraints take simple forms. If function is determined by a goal directed process that favours versatility (e.g. natural selection), then efficiency demands weak constraints take simple forms. Complexity has no causal influence on generalisation, but appears to due to confounding.

Is Complexity an Illusion?

TL;DR

This work interrogates whether complexity causally drives generalisation or is merely an artefact of abstraction. Using a pancomputational, abstraction‑layer formalism with a finite vocabulary, it shows that without abstraction all behaviours have equivalent apparent complexity, making complexity seem illusory; when vocabularies are finite, weakness can masquerade as simplicity and thus correlate with sample efficiency. The authors argue that abstraction is goal-directed, biasing weaker policies to simpler forms under constraints, which explains observed correlations between simplicity and generalisation. The results clarify when complexity-based predictions are meaningful for interactive learning and suggest that observed gains stem from interpretation and constraint structure rather than a fundamental link between complexity and intelligence.

Abstract

Simplicity is held by many to be the key to general intelligence. Simpler models tend to "generalise", identifying the cause or generator of data with greater sample efficiency. The implications of the correlation between simplicity and generalisation extend far beyond computer science, addressing questions of physics and even biology. Yet simplicity is a property of form, while generalisation is of function. In interactive settings, any correlation between the two depends on interpretation. In theory there could be no correlation and yet in practice, there is. Previous theoretical work showed generalisation to be a consequence of "weak" constraints implied by function, not form. Experiments demonstrated choosing weak constraints over simple forms yielded a 110-500% improvement in generalisation rate. Here we show that all constraints can take equally simple forms, regardless of weakness. However if forms are spatially extended, then function is represented using a finite subset of forms. If function is represented using a finite subset of forms, then we can force a correlation between simplicity and generalisation by making weak constraints take simple forms. If function is determined by a goal directed process that favours versatility (e.g. natural selection), then efficiency demands weak constraints take simple forms. Complexity has no causal influence on generalisation, but appears to due to confounding.
Paper Structure (10 sections, 2 theorems, 1 equation)

This paper contains 10 sections, 2 theorems, 1 equation.

Table of Contents

  1. Introduction
  2. What exactly is complexity supposed to indicate?:
  3. Key questions:
  4. Results:
  5. The Formalism
  6. Learning and inference definitions:
  7. Arguments and Results
  8. Implications for Complexity
  9. Time, space and causal confounding:
  10. Why confounding tends to occur:

Key Result

proposition thmcounterproposition

If there is no abstraction, complexity can always be minimized without improving sample efficiency, regardless of the task.

Theorems & Definitions (9)

  • definition thmcounterdefinition: environment
  • definition thmcounterdefinition: abstraction layer
  • definition thmcounterdefinition: $\mathfrak{v}$-task
  • definition thmcounterdefinition: inference
  • definition thmcounterdefinition: learning
  • proposition thmcounterproposition: subjectivity
  • proof
  • proposition thmcounterproposition: confounding
  • proof