Incoherence in goal-conditioned autoregressive models
Jacek Karwowski, Raymond Douglas
TL;DR
This work analyzes incoherence arising in goal-conditioned autoregressive reinforcement learning, where policies conditioned on outcomes can misalign with their own future rollouts. It formalizes incoherence via soft Q/V and an information-theoretic KL measure, and shows that retraining on the agent's own actions monotonically improves return and converges to a coherent policy under suitable conditions. The authors unify three common strategies to remove incoherence—retraining on trajectories, lowering the temperature, and folding the posterior into the reward—showing equivalence in deterministic environments and illuminating the training–inference trade-off through the lens of effective horizon. They provide theoretical results on convergence and equivalence, and discuss implications for training in multi-step, stochastic settings, with potential applications to language-model fine-tuning and RLHF-like techniques. The work lays a principled foundation for aligning goal-conditioned autoregressive policies with their own deployment behavior and suggests directions for practical algorithms and future experiments.
Abstract
We investigate mathematically the notion of incoherence: a structural issue with reinforcement learning policies derived by naive goal-conditioning of autoregressive models. We focus on the process of re-training models on their own actions, that is, fine-tuning offline-learned policies with online RL. We prove that it decreases incoherence and leads to an improvement in return, and we aim to characterize the resulting trajectory of policies. By re-framing standard notions of control-as-inference and soft Q learning, we establish a three-way correspondence with two other ways of understanding the iterative re-training process: as folding the posterior into the reward and, in the deterministic case, as decreasing the temperature parameter; the correspondence has computational content via the training-inference trade-off. Through soft-conditioning generative models, we discuss the link between incoherence and the effective horizon.