Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TempoFormer: A Transformer for Temporally-aware Representations in Change Detection

Talia Tseriotou, Adam Tsakalidis, Maria Liakata

TL;DR

The first task-agnostic transformer-based and temporally-aware model for dynamic representation learning, jointly trained on inter and intra context dynamics and introduces a novel temporal variation of rotary positional embeddings is introduced.

Abstract

Dynamic representation learning plays a pivotal role in understanding the evolution of linguistic content over time. On this front both context and time dynamics as well as their interplay are of prime importance. Current approaches model context via pre-trained representations, which are typically temporally agnostic. Previous work on modelling context and temporal dynamics has used recurrent methods, which are slow and prone to overfitting. Here we introduce TempoFormer, the first task-agnostic transformer-based and temporally-aware model for dynamic representation learning. Our approach is jointly trained on inter and intra context dynamics and introduces a novel temporal variation of rotary positional embeddings. The architecture is flexible and can be used as the temporal representation foundation of other models or applied to different transformer-based architectures. We show new SOTA performance on three different real-time change detection tasks.

TempoFormer: A Transformer for Temporally-aware Representations in Change Detection

TL;DR

The first task-agnostic transformer-based and temporally-aware model for dynamic representation learning, jointly trained on inter and intra context dynamics and introduces a novel temporal variation of rotary positional embeddings is introduced.

Abstract

Dynamic representation learning plays a pivotal role in understanding the evolution of linguistic content over time. On this front both context and time dynamics as well as their interplay are of prime importance. Current approaches model context via pre-trained representations, which are typically temporally agnostic. Previous work on modelling context and temporal dynamics has used recurrent methods, which are slow and prone to overfitting. Here we introduce TempoFormer, the first task-agnostic transformer-based and temporally-aware model for dynamic representation learning. Our approach is jointly trained on inter and intra context dynamics and introduces a novel temporal variation of rotary positional embeddings. The architecture is flexible and can be used as the temporal representation foundation of other models or applied to different transformer-based architectures. We show new SOTA performance on three different real-time change detection tasks.
Paper Structure (27 sections, 4 equations, 3 figures, 15 tables)

This paper contains 27 sections, 4 equations, 3 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Problem Formulation
  5. TempoFormer Overview
  6. Post-level Encoding (Local)
  7. Stream-level Encoding (Global)
  8. Temporal Rotary Multi-Head Attention
  9. Context-enhanced Encoding
  10. Network Fine-Tuning
  11. Experiments
  12. Tasks and Datasets
  13. Baselines and Experimental Setup
  14. Results and Discussion
  15. Comparison against baselines
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Paraphrased example from the task of identifying moments of change in individuals' mood tsakalidis2022identifying. Here, the historical content (light grey) provides important linguistic context towards identifying a Switch, a sudden mood shift from positive to negative, in the user's behaviour at the last post (black).
  • Figure 2: TempoFormer Architecture on 5-post window.
  • Figure 3: TempoFormer Results for Different Contextual Window Sizes.