Incorporating Cognitive Biases into Reinforcement Learning for Financial Decision-Making

TL;DR

The paper addresses incorporating cognitive biases into reinforcement learning for financial decision-making. It proposes a bias-aware RL framework using biased rewards, e.g., loss aversion with $r_{\text{LA}}(s,a)=r(s,a)$ for $r(s,a)\ge0$ and $r_{\text{LA}}(s,a)=\lambda r(s,a)$ for $r(s,a)<0$, and adaptive exploration to model overconfidence. It evaluates a tabular Q-learning agent in a synthetic random-walk market, reporting predominantly negative profitability and stability issues, highlighting the difficulty of achieving benefits from naïve bias integration. The findings emphasize the need for richer bias models and more advanced RL methods, and they provide guidance for future work in realistic environments and potential multi-agent setups.

Abstract

Financial markets are influenced by human behavior that deviates from rationality due to cognitive biases. Traditional reinforcement learning (RL) models for financial decision-making assume rational agents, potentially overlooking the impact of psychological factors. This study integrates cognitive biases into RL frameworks for financial trading, hypothesizing that such models can exhibit human-like trading behavior and achieve better risk-adjusted returns than standard RL agents. We introduce biases, such as overconfidence and loss aversion, into reward structures and decision-making processes and evaluate their performance in simulated and real-world trading environments. Despite its inconclusive or negative results, this study provides insights into the challenges of incorporating human-like biases into RL, offering valuable lessons for developing robust financial AI systems.

Figures (9)

• Figure 1: Hyperparameter tuning results for state space discretization ($n \in \{5, 10, 15, 20\}$). (Left) Sharpe Ratio evolution over epochs. (Right) Cumulative Returns over training epochs. All configurations show negative cumulative returns with high variability, indicating difficulty in learning profitable strategies.
• Figure 2: Loss aversion ablation study with multipliers $\lambda \in \{1, 1.5, 2, 2.5, 3\}$. (Left) Sharpe Ratio trajectories over epochs. (Right) Cumulative Returns evolution. Higher multipliers ($\lambda \geq 2.5$) show worse performance, suggesting that excessive loss aversion amplification degrades learning dynamics.
• Figure 3: Discount factor ablation with $\gamma \in \{0.5, 0.7, 0.9, 0.99\}$. (Left) Sharpe Ratios over epochs. (Right) Cumulative Returns. No clear superiority of any discount factor, with all configurations exhibiting similar volatile performance patterns.
• Figure 4: Action space reduction study (removing "hold" action). (Left) Sharpe Ratios for different state space sizes. (Right) Returns over epochs. Mixed outcomes suggest that action space design significantly impacts learning, with optimal configuration depending on state representation.
• Figure 5: Reward structure variants comparison. (Left) Sharpe Ratio comparison. (Right) Cumulative Returns. Proportional gains (Reward Struct 1) show stable but near-zero returns, while volatility-penalized rewards (Reward Struct 2) exhibit high volatility with negative outcomes.

Theorems & Definitions (12)

• Theorem A.1: Convergence of Q-Learning with Loss-Averse Rewards
• proof
• Theorem A.2: Policy Shift Under Loss Aversion
• proof
• Proposition A.3: Value Function Transformation
• proof
• Proposition A.4: Exploration Rate Impact
• proof
• Proposition A.5: Sharpe Ratio Bound
• proof