FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Finance

TL;DR

FinFlowRL tackles robust stochastic control in non-stationary finance by combining imitation learning over a mixture of expert strategies with a PPO-based fine-tuning stage in a latent noise space. The framework uses action chunking and a frozen MeanFlow expert to generate smooth, multi-step actions at millisecond-scale latency, enabling real-time applicability in high-frequency market making. Empirical results in simulated and historical-market data show FinFlowRL outperforms individually optimized experts across diverse regimes, with notable improvements in PnL and risk metrics and favorable latency and parameter-efficiency characteristics. This work demonstrates a successful cross-disciplinary transfer of flow-based generative policies from robotics to finance and suggests broad applicability to other fast-changing financial decision problems.

Abstract

Traditional stochastic control methods in finance struggle in real world markets due to their reliance on simplifying assumptions and stylized frameworks. Such methods typically perform well in specific, well defined environments but yield suboptimal results in changed, non stationary ones. We introduce FinFlowRL, a novel framework for financial optimal stochastic control. The framework pretrains an adaptive meta policy learning from multiple expert strategies, then finetunes through reinforcement learning in the noise space to optimize the generative process. By employing action chunking generating action sequences rather than single decisions, it addresses the non Markovian nature of markets. FinFlowRL consistently outperforms individually optimized experts across diverse market conditions.

Figures (5)

• Figure 1: MeanFlow-PPO architecture with FlowRL-inspired design. The state $s_t = (t, X_t, q_t, S_t, \text{spread}_t)$ represents time, cash position, inventory, mid-price, and bid-ask spread. The frozen MeanFlow expert (orange, dashed) provides structured action generation while lightweight trainable networks (green) learn task-specific adaptations for optimal bid-ask spread placement.
• Figure 2: MeanFlow transformation from noise space to action space. The learned noise policy generates points in $\mathcal{W}$ that are transformed through the frozen MeanFlow expert to produce smooth action trajectories in $\mathcal{A}$.
• Figure 3: Illustration of MeanFlow velocity fields. Left: The average velocity $u(z,r,t)$ connects points between times $r$ and $t$, while the instantaneous velocity $v$ represents the tangent direction. Right: The average velocity field $u(z,r,t)$ at different time points $t \in \{0.5, 0.7, 1.0\}$, showing how the field evolves over time.
• Figure 4: Hierarchical temporal structure in MeanFlow-PPO. The model observes $T_{obs}$ past states, generates actions for $T_{pred}$ future steps, but only executes the first $T_{exec}$ actions before replanning.
• Figure 5: Flow-State Representation Learning in MeanFlow-PPO. The noise policy learns to transform standard Gaussian noise into task-specific noise distributions that, when passed through the frozen MeanFlow expert, produce optimal trading actions.