Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Deep Reinforcement Learning Framework For Financial Portfolio Management

Jinyang Li

TL;DR

The paper tackles continuous multi-asset portfolio optimization with a model-free deep reinforcement learning framework. It introduces the Ensemble of Identical Independent Evaluators (EIIE) architecture, augmented by a Portfolio-Vector Memory (PVM) and an Online Stochastic Batch Learning (OSBL) scheme, and evaluates CNN, RNN, and LSTM policy networks under realistic transaction-cost constraints. Across cryptocurrency data, the approach yields superior returns and robust risk metrics, while stock-market application shows results near equal-weight baselines, highlighting market-specific effectiveness. The work advances online, data-efficient reinforcement learning for portfolio management and demonstrates practical deployment considerations such as online updating and cost-aware trading.

Abstract

In this research paper, we investigate into a paper named "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem" [arXiv:1706.10059]. It is a portfolio management problem which is solved by deep learning techniques. The original paper proposes a financial-model-free reinforcement learning framework, which consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. Three different instants are used to realize this framework, namely a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). The performance is then examined by comparing to a number of recently reviewed or published portfolio-selection strategies. We have successfully replicated their implementations and evaluations. Besides, we further apply this framework in the stock market, instead of the cryptocurrency market that the original paper uses. The experiment in the cryptocurrency market is consistent with the original paper, which achieve superior returns. But it doesn't perform as well when applied in the stock market.

A Deep Reinforcement Learning Framework For Financial Portfolio Management

TL;DR

The paper tackles continuous multi-asset portfolio optimization with a model-free deep reinforcement learning framework. It introduces the Ensemble of Identical Independent Evaluators (EIIE) architecture, augmented by a Portfolio-Vector Memory (PVM) and an Online Stochastic Batch Learning (OSBL) scheme, and evaluates CNN, RNN, and LSTM policy networks under realistic transaction-cost constraints. Across cryptocurrency data, the approach yields superior returns and robust risk metrics, while stock-market application shows results near equal-weight baselines, highlighting market-specific effectiveness. The work advances online, data-efficient reinforcement learning for portfolio management and demonstrates practical deployment considerations such as online updating and cost-aware trading.

Abstract

In this research paper, we investigate into a paper named "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem" [arXiv:1706.10059]. It is a portfolio management problem which is solved by deep learning techniques. The original paper proposes a financial-model-free reinforcement learning framework, which consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. Three different instants are used to realize this framework, namely a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). The performance is then examined by comparing to a number of recently reviewed or published portfolio-selection strategies. We have successfully replicated their implementations and evaluations. Besides, we further apply this framework in the stock market, instead of the cryptocurrency market that the original paper uses. The experiment in the cryptocurrency market is consistent with the original paper, which achieve superior returns. But it doesn't perform as well when applied in the stock market.
Paper Structure (43 sections, 25 equations, 7 figures, 5 tables)

This paper contains 43 sections, 25 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Background
  4. Portfolio Management Theory
  5. Cryptocurrencies
  6. Reinforcement Learning
  7. Related Work
  8. Problem Definition
  9. Two Assumptions
  10. Mathematical Formalism
  11. Transaction Costs
  12. Data Structure
  13. Asset Selection
  14. Price Tensor
  15. Reinforcement Learning Framework
  16. ...and 28 more sections

Figures (7)

  • Figure 1: The Typical Framing of a Reinforcement Learning (RL) scenario
  • Figure 2: Illustration of trading between periods.
  • Figure 3: CNN Implementation of the EIIE
  • Figure 4: RNN (or LSTM) Implementation of the EIIE
  • Figure 5: Log-scale accumulated wealth from 2017-05-04 to 2017-07-01 - Back-test 1
  • ...and 2 more figures