Subjective functions

TL;DR

The paper argues that a single fixed objective may be insufficient for modeling intelligent behavior and proposes subjective functions—endogenous objective signals defined by the agent itself. It formalizes expected prediction error (EPE) as a concrete subjective objective that drives both inner-loop policy optimization and outer-loop goal selection, enabling open-ended learning. It ties EPE to psychological and neuroscientific findings on prediction errors, hedonic dynamics, and information seeking, and connects it to ML concepts like intrinsic motivation, learning progress, and meta-learning. While not yet a fully operational algorithm, the framework offers a coherent path toward agents that autonomously generate and pursue meaningful, progressively challenging goals.

Abstract

Where do objective functions come from? How do we select what goals to pursue? Human intelligence is adept at synthesizing new objective functions on the fly. How does this work, and can we endow artificial systems with the same ability? This paper proposes an approach to answering these questions, starting with the concept of a subjective function, a higher-order objective function that is endogenous to the agent (i.e., defined with respect to the agent's features, rather than an external task). Expected prediction error is studied as a concrete example of a subjective function. This proposal has many connections to ideas in psychology, neuroscience, and machine learning.