What Teaches Robots to Walk, Teaches Them to Trade too -- Regime Adaptive Execution using Informed Data and LLMs

TL;DR

An innovative approach that leverages world knowledge of pretrained LLMs and dynamically adapts them using intrinsic, natural market rewards using LLM alignment technique that demonstrates the efficacy of this method in adapting to regime shifts in financial markets.

Abstract

Machine learning techniques applied to the problem of financial market forecasting struggle with dynamic regime switching, or underlying correlation and covariance shifts in true (hidden) market variables. Drawing inspiration from the success of reinforcement learning in robotics, particularly in agile locomotion adaptation of quadruped robots to unseen terrains, we introduce an innovative approach that leverages world knowledge of pretrained LLMs (aka. 'privileged information' in robotics) and dynamically adapts them using intrinsic, natural market rewards using LLM alignment technique we dub as "Reinforcement Learning from Market Feedback" (**RLMF**). Strong empirical results demonstrate the efficacy of our method in adapting to regime shifts in financial markets, a challenge that has long plagued predictive models in this domain. The proposed algorithmic framework outperforms best-performing SOTA LLM models on the existing (FLARE) benchmark stock-movement (SM) tasks by more than 15\% improved accuracy. On the recently proposed NIFTY SM task, our adaptive policy outperforms the SOTA best performing trillion parameter models like GPT-4. The paper details the dual-phase, teacher-student architecture and implementation of our model, the empirical results obtained, and an analysis of the role of language embeddings in terms of Information Gain.

Figures (8)

• Figure 1: We propose Regime Adaptive Execution in the financial market setting motivated by the success of reinforcement learning inspired robust locomotion methods supplanting intricate heuristic control architectures in quadrupedal robots, thereby eschewing decades-old conventional heuristic approaches to the 'market regime classification problem'.
• Figure 2: Robot locomotion: high-level schematic of common dual-policy SOTA approaches.
• Figure 3: A snapshot of the 'news' key value on date: 2020-02-06, at the upstart of the global coronavirus epidemic. Our $\pi_{LM}$ policy's prompt is composed of task instruction as query prefix, market context, and this news value concatenated: $s.t.$$x_p \leftarrow (x_{instruction}; x_{context}; x_{news})$. The semantic text colors red, and green conveys negative and positive sentiments. The day's market relevent news was dominated by mostly negative sentiments.
• Figure 4: Breaking down the instruction or prompt prefix, and market context components of a prompt, $x_p$.
• Figure 5: Regime adaptive execution uses the NIFTY dataset to train a reward model (RM) and align a pretrained LLM during the training phase. In the deployment phase, streaming online market data is used to continually update the RM, subsequently a student policy that swaps place with an executor teacher policy after windowed intervals.