Retrieval-Augmented Diffusion Models for Time Series Forecasting

TL;DR

The proposed Retrieval- Augmented Time series Diffusion model allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets, and compensates for the deficiencies of existing time series diffusion models in terms of guidance.

Abstract

While time series diffusion models have received considerable focus from many recent works, the performance of existing models remains highly unstable. Factors limiting time series diffusion models include insufficient time series datasets and the absence of guidance. To address these limitations, we propose a Retrieval- Augmented Time series Diffusion model (RATD). The framework of RATD consists of two parts: an embedding-based retrieval process and a reference-guided diffusion model. In the first part, RATD retrieves the time series that are most relevant to historical time series from the database as references. The references are utilized to guide the denoising process in the second part. Our approach allows leveraging meaningful samples within the database to aid in sampling, thus maximizing the utilization of datasets. Meanwhile, this reference-guided mechanism also compensates for the deficiencies of existing time series diffusion models in terms of guidance. Experiments and visualizations on multiple datasets demonstrate the effectiveness of our approach, particularly in complicated prediction tasks.

Figures (6)

• Figure 1: (a) The figure shows the differences in forecasting results between the CSDI tashiro2021csdi (left) and RATD(right). Due to the very small proportion of such cases in the training set, CSDI struggles to make accurate predictions, often predicting more common results. Our method, by retrieving meaningful references as guidance, makes much more accurate predictions. (b) A comparison between our method's framework(bottom) and the conventional time series diffusion model framework(top). (c) We randomly selected 25 forecasting tasks from the electricity dataset. Compared to our method, CSDI and MG-TSD fan2024mg exhibited significantly higher instability. This indicates that the RATD is better at handling complex tasks that are challenging for the other two methods.
• Figure 2: Overview of the proposed RATD. The historical time series ${\bm{x}}^H$ is inputted into the retrieval module to for the corresponding references ${\bm{x}}^R$. After that, ${\bm{x}}^H$ is concatenated with the noise as the main input for the model $\mu_{\theta}$. ${\bm{x}}^R$ will be utilized as the guidance for the denoising process.
• Figure 3: The structure of $\mu_{\theta}$. (a) The main architecture of $\mu_{\theta}$ is the time series transformer structure that proved effective. (b) The structure of the proposed RMA. We integrate three different features through matrix multiplication.
• Figure 4: Visualizations on wind by CSDI, $\text{D}_3$VAE, iTransformer and the proposed RATD (with reference).
• Figure 5: The effect of hyper-parameter $n$ and $k$.