The Limits of Complexity: Why Feature Engineering Beats Deep Learning in Investor Flow Prediction

TL;DR

The paper questions the premise that deeper ML architectures outperform engineered features in financial prediction, using a Korean investor-flow dataset (2,439 stocks, 2,788,940 observations; 2020–2024). It applies market-cap normalization (Matched Filter), Independent Component Analysis, Wavelet Coherence, and LSTM with attention to predict next-day returns. The key finding is that a simple linear model on normalized flows delivers a Sharpe of $1.30$ and cumulative return of $272.6\%$, while the full ICA–Wavelet–LSTM pipeline yields a Sharpe of $0.07$ and a cumulative return of $-5.1\%$, with the LSTM collapsing to the unconditional mean. The work identifies boundary conditions for ML in finance, showing that feature engineering can dominate, and documents specific failure modes of complex models in low-SNR, non-stationary markets, offering practical guidance for practitioners and researchers to calibrate expectations and methods.

Abstract

The application of machine learning to financial prediction has accelerated dramatically, yet the conditions under which complex models outperform simple alternatives remain poorly understood. This paper investigates whether advanced signal processing and deep learning techniques can extract predictive value from investor order flows beyond what simple feature engineering achieves. Using a comprehensive dataset of 2.79 million observations spanning 2,439 Korean equities from 2020--2024, we apply three methodologies: \textit{Independent Component Analysis} (ICA) to recover latent market drivers, \textit{Wavelet Coherence} analysis to characterize multi-scale correlation structure, and \textit{Long Short-Term Memory} (LSTM) networks with attention mechanisms for non-linear prediction. Our results reveal a striking finding: a parsimonious linear model using market capitalization-normalized flows (``Matched Filter'' preprocessing) achieves a Sharpe ratio of 1.30 and cumulative return of 272.6\%, while the full ICA-Wavelet-LSTM pipeline generates a Sharpe ratio of only 0.07 with a cumulative return of $-5.1\%$. The raw LSTM model collapsed to predicting the unconditional mean, achieving a hit rate of 47.5\% -- worse than random. We conclude that in low signal-to-noise financial environments, domain-specific feature engineering yields substantially higher marginal returns than algorithmic complexity. These findings establish important boundary conditions for the application of deep learning to financial prediction.

Figures (9)

• Figure 1: Independent Component Analysis (ICA) of Korean investor flows. Figure (a) shows the time-series of the three extracted components, while (b) illustrates the loading of each investor type onto these components.
• Figure 2: Correlations between ICA components and external market factors (KOSPI returns, VIX, USD/KRW, etc.). IC$_1$ shows unexpectedly high correlation with market returns rather than macro factors.
• Figure 3: Wavelet Coherence between Foreign and Institutional investor flows. The coherence increases monotonically with scale, indicating stronger synchronization at fundamental rather than high-frequency horizons.
• Figure 4: Multi-scale clustering of investor flow correlations. The results show a clear separation between high-frequency noise and lower-frequency fundamental alignment.
• Figure 5: LSTM Prediction Performance. The scatter plot of predicted vs realized returns show a horizontal line at the mean, illustrating the model's collapse to predicting the unconditional mean due to low SNR.