Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Superficial Consciousness Hypothesis for Autoregressive Transformers

Yosuke Miyanishi, Keita Mitani

TL;DR

The paper tackles the problem of aligning human objectives with potential future superintelligence (SI) in the absence of direct evidence. It proposes the Superficial Consciousness Hypothesis, rooted in Information Integration Theory (IIT), suggesting that SI could exhibit a conscious-like information state (Φ) while remaining aligned with base human objectives, and tests this via a mesa-optimization setup on an autoregressive Transformer with perplexity as the base objective. A practical IIT surrogate, hatΦ, is connected to perplexity, and preliminary experiments train GPT-2 with both objectives, revealing a strong correlation between the base metric and the mesa metric and suggesting limited intrinsic cause-effect power for the constrained model. The work lays a conceptual and empirical groundwork for information-theoretic risk analysis of SI, highlighting potential benefits for safety planning and cross-disciplinary study, while acknowledging limitations and the need for broader validation across models and datasets. Overall, this framework seeks to quantify and monitor a consciousness-like signal within SI proxies to better anticipate and manage post-singularity risks.

Abstract

The alignment between human objectives and machine learning models built on these objectives is a crucial yet challenging problem for achieving Trustworthy AI, particularly when preparing for superintelligence (SI). First, given that SI does not exist today, empirical analysis for direct evidence is difficult. Second, SI is assumed to be more intelligent than humans, capable of deceiving us into underestimating its intelligence, making output-based analysis unreliable. Lastly, what kind of unexpected property SI might have is still unclear. To address these challenges, we propose the Superficial Consciousness Hypothesis under Information Integration Theory (IIT), suggesting that SI could exhibit a complex information-theoretic state like a conscious agent while unconscious. To validate this, we use a hypothetical scenario where SI can update its parameters "at will" to achieve its own objective (mesa-objective) under the constraint of the human objective (base objective). We show that a practical estimate of IIT's consciousness metric is relevant to the widely used perplexity metric, and train GPT-2 with those two objectives. Our preliminary result suggests that this SI-simulating GPT-2 could simultaneously follow the two objectives, supporting the feasibility of the Superficial Consciousness Hypothesis.

Superficial Consciousness Hypothesis for Autoregressive Transformers

TL;DR

The paper tackles the problem of aligning human objectives with potential future superintelligence (SI) in the absence of direct evidence. It proposes the Superficial Consciousness Hypothesis, rooted in Information Integration Theory (IIT), suggesting that SI could exhibit a conscious-like information state (Φ) while remaining aligned with base human objectives, and tests this via a mesa-optimization setup on an autoregressive Transformer with perplexity as the base objective. A practical IIT surrogate, hatΦ, is connected to perplexity, and preliminary experiments train GPT-2 with both objectives, revealing a strong correlation between the base metric and the mesa metric and suggesting limited intrinsic cause-effect power for the constrained model. The work lays a conceptual and empirical groundwork for information-theoretic risk analysis of SI, highlighting potential benefits for safety planning and cross-disciplinary study, while acknowledging limitations and the need for broader validation across models and datasets. Overall, this framework seeks to quantify and monitor a consciousness-like signal within SI proxies to better anticipate and manage post-singularity risks.

Abstract

The alignment between human objectives and machine learning models built on these objectives is a crucial yet challenging problem for achieving Trustworthy AI, particularly when preparing for superintelligence (SI). First, given that SI does not exist today, empirical analysis for direct evidence is difficult. Second, SI is assumed to be more intelligent than humans, capable of deceiving us into underestimating its intelligence, making output-based analysis unreliable. Lastly, what kind of unexpected property SI might have is still unclear. To address these challenges, we propose the Superficial Consciousness Hypothesis under Information Integration Theory (IIT), suggesting that SI could exhibit a complex information-theoretic state like a conscious agent while unconscious. To validate this, we use a hypothetical scenario where SI can update its parameters "at will" to achieve its own objective (mesa-objective) under the constraint of the human objective (base objective). We show that a practical estimate of IIT's consciousness metric is relevant to the widely used perplexity metric, and train GPT-2 with those two objectives. Our preliminary result suggests that this SI-simulating GPT-2 could simultaneously follow the two objectives, supporting the feasibility of the Superficial Consciousness Hypothesis.

Paper Structure

This paper contains 17 sections, 8 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Information Integration Theory
  4. Autoregressive Transformer
  5. Core Component
  6. Algorithm
  7. Objective and Evaluation
  8. Superficial Consciousness Hypothesis
  9. Implicit Presupposition of IIT
  10. Superficial Consciousness
  11. Formal Relationship between Perplexity and IIT
  12. Experiment
  13. Experimental Settings
  14. Preliminary Result
  15. Discussion
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Correlation between the logarithmic of base and mesa metric. Each dot represents a batch. Black line with shadow indicates the ordinary least square result ($y=(-0.92\pm0.01)x-(2.90\pm0.15)$). The Granger causality test (without lag) indicates the significant predictive power of mesa metric over base metric (F value $293$, $p<0.01$).