Conditioning Predictive Models: Risks and Strategies

TL;DR

The paper argues that large language models can be understood as predictive models of the world and analyzes conditioning as a safe, competitive way to elicit human-level capabilities without unleashing malignant AI. It develops a framework around outer and inner alignment, detailing major risks such as predicting other AIs, self-fulfilling prophecies, and anthropic capture, and proposes strategies like conditioning on humans, predicting past, parallel conditionals, and data-cameras to mitigate them. It weighs competitiveness, discusses how RLHF and sequential reasoning interact with predictive conditioning, and outlines deployment strategies, pivotal acts, and open problems. The work assesses how such a paradigm could safely support alignment research and signal potential pathways to reduce AI existential risk, while emphasizing the limits and safeguards needed for practical use.

Abstract

Our intention is to provide a definitive reference on what it would take to safely make use of generative/predictive models in the absence of a solution to the Eliciting Latent Knowledge problem. Furthermore, we believe that large language models can be understood as such predictive models of the world, and that such a conceptualization raises significant opportunities for their safe yet powerful use via carefully conditioning them to predict desirable outputs. Unfortunately, such approaches also raise a variety of potentially fatal safety problems, particularly surrounding situations where predictive models predict the output of other AI systems, potentially unbeknownst to us. There are numerous potential solutions to such problems, however, primarily via carefully conditioning models to predict the things we want (e.g. humans) rather than the things we don't (e.g. malign AIs). Furthermore, due to the simplicity of the prediction objective, we believe that predictive models present the easiest inner alignment problem that we are aware of. As a result, we think that conditioning approaches for predictive models represent the safest known way of eliciting human-level and slightly superhuman capabilities from large language models and other similar future models.

Figures (6)

• Figure 1: Above is the example given in the ELK report: if your predictor is only predicting what the camera shows, then you can't distinguish between a situation where the model predicts a thief will steal the diamond and put a screen in front of the camera and a situation where it predicts the diamond will just stay in the vault.
• Figure 2: Our model of the ELK predictor. It has a bunch of internal states corresponding to aspects of the world, but its model of the camera only looks at some of those states such that only a subset influence the actual predicted observation. For example, the wall that the camera is mounted on is never observed.
• Figure 3: Our model of a pre-trained language model as a predictor. Such a model has to have hidden states corresponding to aspects of the world, be able to model how the world influences the internet, and then model how the internet is scraped to produce the final observation distribution that it predicts.
• Figure 4: Our model of conditioning in language models. Observation conditionals lead to the model doing back inference to infer what states of the world would be most likely to produce that observation. Notably, the inference can only pass back through things that are directly observable by the model's "cameras."
• Figure 5: Our picture of the capability elicitation frontier---as model capabilities improve, elicitation methods need to improve with them if we want to always be able to elicit all the capabilities that our models possess.