REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning

TL;DR

This work proposes a novel method of using a learned measure, the Retrieval-Based Reconstruction (REBAR) measure, that measures the similarity between two sequences as the reconstruction error that results from reconstructing one sequence with retrieved information from the other.

Abstract

The success of self-supervised contrastive learning hinges on identifying positive data pairs, such that when they are pushed together in embedding space, the space encodes useful information for subsequent downstream tasks. Constructing positive pairs is non-trivial as the pairing must be similar enough to reflect a shared semantic meaning, but different enough to capture within-class variation. Classical approaches in vision use augmentations to exploit well-established invariances to construct positive pairs, but invariances in the time-series domain are much less obvious. In our work, we propose a novel method of using a learned measure for identifying positive pairs. Our Retrieval-Based Reconstruction (REBAR) measure measures the similarity between two sequences as the reconstruction error that results from reconstructing one sequence with retrieved information from the other. Then, if the two sequences have high REBAR similarity, we label them as a positive pair. Through validation experiments, we show that the REBAR error is a predictor of mutual class membership. Once integrated into a contrastive learning framework, our REBAR method learns an embedding that achieves state-of-the-art performance on downstream tasks across various modalities.

Figures (23)

• Figure 1: This figure demonstrates the intuition of our Retrieval-Based Reconstruction (REBAR) approach. If we can successfully retrieve information from another subsequence to aid in reconstruction, then the two subsequences should form a positive pair in a contrastive learning framework. We first use the context-window, designated by the grey box, of Walk Subseq 1 to query for information in Walk Subseq 2 or in Sit Subseq. Upper b) shows that the context window in Walk Subseq 2 provides a good match with a similar double peak motif, leading to a good reconstruction. Lower b) shows that Sit Subseq has no matching motif, leading to a poor reconstruction.
• Figure 2: 1) First, our REBAR cross-attention is trained to retrieve information from the key to reconstruct a masked-out query. 2) Next, it is frozen and utilized to identify the positive instance. After sampling subsequences from the time-series, the subsequence that reconstructed the anchor with the lowest REBAR error is labeled as positive, and the others are labeled as within-time-series negatives. These negatives capture how time-series dynamics can change over time. Subsequences from other time-series within a data batch are labeled as between-time-series-negatives, and these negatives capture differences among patients. 3) We use the assigned labels to train an encoder.
• Figure 3: REBAR Cross-attention reconstruction of a single masked-out point $\mathaccentV{bar}0\symboldoperators16{\textbf{x}}_q$, $\textrm{REBAR}(\mathaccentV{bar}0\symboldoperators16{\textbf{x}}_q ; {\textbf{X}_k})$. Red designates the functions used for the attention weight calculation. In 1), the attention weights identify which region in $\textbf{X}_k$ should be retrieved for reconstruction by comparing the unmasked context around $\mathaccentV{bar}0\symboldoperators16{\textbf{x}}_q$ via the $f_q$ dilated convolution, with the motifs in $\textbf{X}_k$ via the $f_k$ dilated convolution. In 2), the attention weights are used to retrieve an $f_v$ transformation of the key in a weighted average for reconstruction. Minor model details (e.g. norms) are omitted for brevity.
• Figure 4: Comparison of different $f_{q/k}$ within $\textrm{sim}( f_q( \mathaccentV{bar}0\symboldoperators16{\textbf{x}}_{q} ), f_k( \textbf{x}_{k}) )$. The REBAR's $f\coloneqq$ Dilated Convolution allows for semantically-meaningful motif comparison within the retrieval function, unlike in the vanilla's $f\coloneqq\textbf{x}\textbf{W}$, in which single time-points are compared with another.
• Figure 5: There is a high concentration on the diagonals of the confusion matrices across all of our datasets. This shows that REBAR, although trained with a reconstruction task without class labels, is able to predict mutual class membership, validating our idea for using REBAR to identify positive pairs in contrastive learning.