Ethereum Price Prediction Employing Large Language Models for Short-term and Few-shot Forecasting

TL;DR

This work investigates Ethereum price forecasting using Large Language Models (LLMs) in short-term and few-shot settings. It adapts pre-trained LLMs (Llama-2, Llama-3, GPT-2) by freezing core transformer blocks and fine-tuning selective components to process univariate price time series, using a 7-day history to predict the next day. Experiments on Kaggle and a Node-derived Ethereum dataset show that LLM backbones achieve state-of-the-art performance across $MSE$, $RMSE$, and $MAE$, with Llama-3 often outperforming GPT-2 and PatchTST, especially in short-term forecasting. The results indicate strong potential for LLM-based time-series forecasting in crypto markets and motivate future work incorporating sentiment signals for further accuracy gains.

Abstract

Cryptocurrencies have transformed financial markets with their innovative blockchain technology and volatile price movements, presenting both challenges and opportunities for predictive analytics. Ethereum, being one of the leading cryptocurrencies, has experienced significant market fluctuations, making its price prediction an attractive yet complex problem. This paper presents a comprehensive study on the effectiveness of Large Language Models (LLMs) in predicting Ethereum prices for short-term and few-shot forecasting scenarios. The main challenge in training models for time series analysis is the lack of data. We address this by leveraging a novel approach that adapts existing pre-trained LLMs on natural language or images from billions of tokens to the unique characteristics of Ethereum price time series data. Through thorough experimentation and comparison with traditional and contemporary models, our results demonstrate that selectively freezing certain layers of pre-trained LLMs achieves state-of-the-art performance in this domain. This approach consistently surpasses benchmarks across multiple metrics, including Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE), demonstrating its effectiveness and robustness. Our research not only contributes to the existing body of knowledge on LLMs but also provides practical insights in the cryptocurrency prediction domain. The adaptability of pre-trained LLMs to handle the nature of Ethereum prices suggests a promising direction for future research, potentially including the integration of sentiment analysis to further refine forecasting accuracy.

Figures (3)

• Figure 1: Visualization of the volatility of Ethereum open price in different timestamps: Kaggle Dataset (a), Node Dataset (b).
• Figure 2: Visualization of the models' architecture and pipeline. The architecture of our model pipeline incorporating Llama-2 (Left) and GPT-2 (Right) for Ethereum price prediction. The pipeline utilizes a combination of frozen and fine-tuned layers. For Llama-2, the grouped query attention and feed-forward layers are frozen, while rotary positional embeddings and RMS normalization layers are fine tuned. In GPT-2, the multi-head attention and feed-forward layers are frozen, while positional embeddings and layer normalization layers are fine-tuned. Data normalization and patching techniques are applied to enhance semantic information extraction. What applies to Llama-2, applies to Llama-3 as well.
• Figure 3: Visualization of Ethereum price predictions depicting the actual and predicted values in both short-term and few-shot forecasting for the Kaggle dataset: (a) GPT2 in short-term forecasting and (b) Llama-3 in short-term forecasting c) GPT2 in Few-shot forecasting and d) Llama-3 in Few-shot forecasting.