A Framework for Empowering Reinforcement Learning Agents with Causal Analysis: Enhancing Automated Cryptocurrency Trading

TL;DR

The paper tackles profitable automated trading in volatile cryptocurrency markets by proposing the CausalReinforceNet (CRN) framework, which combines Bayesian-network–based feature engineering with dynamic Bayesian network–driven price-direction signals to augment reinforcement learning traders. Two model-free RL agents (PPO and DDPG) are deployed within CRN, leveraging curated state representations and adaptive position sizing under risk controls, and are evaluated against Buy-and-Hold and a baseline OHLCV RL model across Binance Coin, Ethereum, Litecoin, Ripple, and Tether. The main contributions include the CRN architecture that fuses causal analysis with RL, per-coin feature selection via BN, DBN-informed observations, and a comparative empirical analysis of two RL algorithms under realistic transaction costs and risk rules. Results show CRN generally improves profitability over baselines, with performance varying by coin and algorithm, and provide insights into decision patterns arising from causal features and price-direction predictions. The work underscores the value of explainable, risk-aware RL frameworks for cryptocurrency trading and points to future extensions such as higher-frequency trading, adaptive thresholds, portfolio-level RL, and broader domain applications.

Abstract

Despite advances in artificial intelligence-enhanced trading methods, developing a profitable automated trading system remains challenging in the rapidly evolving cryptocurrency market. This research focuses on developing a reinforcement learning (RL) framework to tackle the complexities of trading five prominent altcoins: Binance Coin, Ethereum, Litecoin, Ripple, and Tether. To this end, we present the CausalReinforceNet~(CRN) framework, which integrates both Bayesian and dynamic Bayesian network techniques to empower the RL agent in trade decision-making. We develop two agents using the framework based on distinct RL algorithms to analyse performance compared to the Buy-and-Hold benchmark strategy and a baseline RL model. The results indicate that our framework surpasses both models in profitability, highlighting CRN's consistent superiority, although the level of effectiveness varies across different cryptocurrencies.

Figures (6)

• Figure 1: RL Classic Cycle -- The agent-environment interaction cycle is a continuous loop where an agent interacts with its environment in discrete time steps. At each time step, the agent observes the current state of the environment and selects an action based on its policy. After taking action, the agent receives a reward and uses it to optimise its policy. The ultimate goal for the agent is to develop a policy that allows it to maximise cumulative rewards over time.
• Figure 2: CRN Architecture -- The framework comprises four essential elements. The first category encompasses input features containing market data, technical indicators, financial assets, and social media. The second element consists of a BN module, which is responsible for the feature engineering of the inputs. The third element is the DBN, which generates daily price directions based on the selected features. Finally, the fourth element represents the automated RL-based trading system.
• Figure 3: Price Plots -- The plots display the price patterns and fluctuations of studied cryptocurrencies between January 2018 and April 2023.
• Figure 4: Performance Comparison Visualisation -- The chart compares the annual ROI of studied cryptocurrencies as detailed in Table \ref{['tab:preds']}.
• Figure 5: $CRN_{PPO}$ Decision Visualisation - These charts provide visualisation of the decisions made by the $CRN_{PPO}$ agent. Sub-figure(a) shows the percentage of each action taken relative to the total number of actions performed, while Sub-figure(b) illustrates the average position size for both sell and buy actions.