On The Expressivity of Objective-Specification Formalisms in Reinforcement Learning
Rohan Subramani, Marcus Williams, Max Heitmann, Halfdan Holm, Charlie Griffin, Joar Skalse
TL;DR
The paper systematically maps the expressive power of 17 objective-specification formalisms for RL beyond Markov rewards by framing expressivity as policy-ordering capabilities and presenting a Hasse diagram over formalisms. It proves several striking equivalences (e.g., INMR=IMORL=FTR; OMORL=FOMR=FTLR; GOMORL=OMO=TLO) and a broad set of non-expressivity results that reveal fundamental incommensurability between formalisms. The work connects these findings to classic theories (VNM) and reward-learning practices, emphasizing that no single formalism universally dominates in expressivity or tractability. It further discusses practical implications for reward learning and objective specification, stressing the need to develop reward-learning methods that can handle a wider variety of formalisms while considering tractability. Overall, this work provides a cohesive reference for selecting and developing RL objective formalisms, guiding future exploration of the expressivity–tractability tradeoffs in policy optimization and reward learning.
Abstract
Most algorithms in reinforcement learning (RL) require that the objective is formalised with a Markovian reward function. However, it is well-known that certain tasks cannot be expressed by means of an objective in the Markov rewards formalism, motivating the study of alternative objective-specification formalisms in RL such as Linear Temporal Logic and Multi-Objective Reinforcement Learning. To date, there has not yet been any thorough analysis of how these formalisms relate to each other in terms of their expressivity. We fill this gap in the existing literature by providing a comprehensive comparison of 17 salient objective-specification formalisms. We place these formalisms in a preorder based on their expressive power, and present this preorder as a Hasse diagram. We find a variety of limitations for the different formalisms, and argue that no formalism is both dominantly expressive and straightforward to optimise with current techniques. For example, we prove that each of Regularised RL, (Outer) Nonlinear Markov Rewards, Reward Machines, Linear Temporal Logic, and Limit Average Rewards can express a task that the others cannot. The significance of our results is twofold. First, we identify important expressivity limitations to consider when specifying objectives for policy optimization. Second, our results highlight the need for future research which adapts reward learning to work with a greater variety of formalisms, since many existing reward learning methods assume that the desired objective takes a Markovian form. Our work contributes towards a more cohesive understanding of the costs and benefits of different RL objective-specification formalisms.