Model-Agnostic Solutions for Deep Reinforcement Learning in Non-Ergodic Contexts

TL;DR

The paper addresses the mismatch between time-average growth and ensemble-averaged rewards in non-ergodic reinforcement learning. It proposes a DRL framework that injects explicit temporal dependence by training over repeated trajectories, avoiding reward-altering transformations. Empirical results show that standard DRL methods fail to learn growth-optimal policies under multiplicative, non-ergodic dynamics unless temporal repetition is introduced; with path-dependent training, Deep Q-Networks begin to align with growth-rate objectives and Actor–Critic models can approach Kelly-optimal strategies. This work offers a practical route to non-ergodic policy learning with potential impact on finance and other domains characterized by multiplicative dynamics, by integrating ergodic reasoning into the training process rather than modifying objectives or rewards.

Abstract

Reinforcement Learning (RL) remains a central optimisation framework in machine learning. Although RL agents can converge to optimal solutions, the definition of ``optimality'' depends on the environment's statistical properties. The Bellman equation, central to most RL algorithms, is formulated in terms of expected values of future rewards. However, when ergodicity is broken, long-term outcomes depend on the specific trajectory rather than on the ensemble average. In such settings, the ensemble average diverges from the time-average growth experienced by individual agents, with expected-value formulations yielding systematically suboptimal policies. Prior studies demonstrated that traditional RL architectures fail to recover the true optimum in non-ergodic environments. We extend this analysis to deep RL implementations and show that these, too, produce suboptimal policies under non-ergodic dynamics. Introducing explicit time dependence into the learning process can correct this limitation. By allowing the network's function approximation to incorporate temporal information, the agent can estimate value functions consistent with the process's intrinsic growth rate. This improvement does not require altering the environmental feedback, such as reward transformations or modified objective functions, but arises naturally from the agent's exposure to temporal trajectories. Our results contribute to the growing body of research on reinforcement learning methods for non-ergodic systems.

Figures (6)

• Figure 1: Illustration of our toy model for training an agent in a multiplicative dynamic \ref{['fig:toy_model']}. An agent needs to decide which of two actions ($a_\text{safe}, a_{risky}$) is most beneficial for optimising its wealth, given a probability $p$ to receive the worst outcome on the risky action. Our proposed solution to manage the non-ergodic context uses repetitions in the action-selection, indicated by the return of the updated wealth to the state for the next time step. Illustration of the two different policies \ref{['fig:genericpolicy']}. The blue solid function describes an optimisation based on expected values and predicts the indifference point $p_{E}$. The green dashed function simulates an agent optimising for growth rates, with the predicted indifference point at $P_{T}$.
• Figure 2: Illustration of the portfolio assignment problem, where the agent needs to find the optimal fraction ($f^{*}$) of its wealth to invest in a portfolio \ref{['fig:kelly_setup']}. Illustration of an expected value policy, solid blue curve, and the optimal policy following the Kelly criterion, dashed green curve, for a policy assignment optimisation problem shown in \ref{['fig:kellygeneric']}.
• Figure 3: Learned policies under varying repetitions $M$ in the toy model. Probability of selecting the safe action as a function of $p$, shown in \ref{['fig:DQN_toymodel']}, the probability of the worst risky outcome—single-step training reproduces expected-value behaviour, while repetitions expose path dependence. The indifference point $p_{0}$ shifts from the expected-value prediction $p_{E}$ (blue, solid) towards the time-growth prediction $p_{T}$ (green, dashed) as $M$ increases \ref{['fig:DQN_policy']}.
• Figure 4: Resulting policy for an actor-critic DRL model trained on optimising the portfolio assignment problem. The mean policy for agents with identical training parameters is shown in \ref{['kelly_mean']}, the median of these policies is shown in \ref{['kelly_median']}. The agents only learn the optimal strategy prescribed by the Kelly Objective when subjected to path-dependent training, as suggested by our alternative training method.
• Figure 5: Analysis of the mean squared error (MSE) between the different policies and the theoretical policies based on the expected value optimisation \ref{['ev_MSE']}, and between the policies and the optimal Kelly Objective \ref{['kelly_MSE']}. The total MSE over the full policy is shown in the legend for each number of repetitions, indicating that, as the number of repetitions increases, the policy aligns more closely with the optimal policy.