History Is Not Enough: An Adaptive Dataflow System for Financial Time-Series Synthesis

TL;DR

The paper tackles the persistent gap between training and real-world performance in quantitative finance caused by concept drift and distributional non-stationarity. It introduces a drift-aware adaptive dataflow that unifies financial data synthesis, augmentation scheduling, and learning-guided feedback control via a bi-level optimization framework. A parameterized data manipulation module enforces financial priors and constraints, while a learning-based planner–scheduler dynamically adjusts augmentation strength and data proportion based on validation signals, enabling provenance-aware replay. Empirical results across forecasting and reinforcement learning trading tasks show improved robustness and risk-adjusted returns, with ablations confirming the necessity of adaptive mixups and curriculum. The proposed framework offers a generalizable, learnable data-management approach that can adapt to evolving markets and support end-to-end learning pipelines in finance.

Abstract

In quantitative finance, the gap between training and real-world performance-driven by concept drift and distributional non-stationarity-remains a critical obstacle for building reliable data-driven systems. Models trained on static historical data often overfit, resulting in poor generalization in dynamic markets. The mantra "History Is Not Enough" underscores the need for adaptive data generation that learns to evolve with the market rather than relying solely on past observations. We present a drift-aware dataflow system that integrates machine learning-based adaptive control into the data curation process. The system couples a parameterized data manipulation module comprising single-stock transformations, multi-stock mix-ups, and curation operations, with an adaptive planner-scheduler that employs gradient-based bi-level optimization to control the system. This design unifies data augmentation, curriculum learning, and data workflow management under a single differentiable framework, enabling provenance-aware replay and continuous data quality monitoring. Extensive experiments on forecasting and reinforcement learning trading tasks demonstrate that our framework enhances model robustness and improves risk-adjusted returns. The system provides a generalizable approach to adaptive data management and learning-guided workflow automation for financial data.

Figures (8)

• Figure 1: t-SNE visualizations to assess concept drift. Top row: t-SNE plot for $P(Y|X)$, where the x-axis represents the feature $X$ and the y-axis represents the daily return as the $Y$. Bottom row: t-SNE plot for $P(X)$. Orange dots mark the data in the training set, while blue dots mark the data in the test set.
• Figure 2: The workflow of training the planner and task model to learn a policy of controlling the data manipulation module with the validation loss of the task model. The training step of the planner is marked with (1), (2), (3), and $f_{\theta^{'}}$ is a copy of $f_{\theta}$. The fire icon marks the flow where parameters are updated.
• Figure 3: The proposed data manipulation module. The manipulated dimension of data is marked with the respective color.
• Figure 4: Trading results where buy and sell actions are marked with green and red labels.
• Figure 5: Operation weights from planner for (a) Transformer and (b) LSTM.