Set2Seq Transformer: Temporal and Positional-Aware Set Representations for Sequential Multiple-Instance Learning

TL;DR

The paper introduces Set2Seq Transformer, a novel architecture for sequential multiple-instance learning that jointly models permutation-invariant set structure within each timestep and temporal dynamics across timesteps. It constructs per-timestep embeddings via a set encoder (DeepSets or Set Transformer) and enriches them with learnable temporal embeddings and positional encodings, producing a final sequence representation processed by a Transformer to predict outcomes from the last timestep. Evaluated on WikiArt-Seq2Rank for artistic-success ranking and Mesogeos for short-term wildfire forecasting, Set2Seq outperforms static MIL and standard temporal baselines, demonstrating cross-domain effectiveness and robustness to distributional shifts. The approach highlights the importance of integrating within-set permutation invariance with explicit temporal structure to capture both local and global patterns in complex sequential data.

Abstract

Sequential multiple-instance learning involves learning representations of sets distributed across discrete timesteps. In many real-world applications, modeling both the internal structure of sets and their temporal relationships across time is essential for capturing complex underlying patterns. However, existing methods either focus on learning set representations at a static level, ignoring temporal dynamics, or treat sequences as ordered lists of individual elements, lacking explicit mechanisms to represent sets. In this work, we propose Set2Seq Transformer, a novel architecture that jointly models permutation-invariant set structure and temporal dependencies by learning temporal and positional-aware representations of sets within a sequence in an end-to-end multimodal manner. We evaluate our Set2Seq Transformer on two tasks that require modeling both set structure alongside temporal and positional patterns, but differ significantly in domain, modality, and objective. First, we consider a fine-art analysis task, modeling artists' oeuvres for predicting artistic success using a novel dataset, WikiArt-Seq2Rank. Second, we utilize our Set2Seq Transformer for a short-term wildfire danger forecasting task. Through extensive experimentation, we show that our Set2Seq Transformer significantly improves over traditional static multiple-instance learning methods by effectively learning permutation-invariant set, temporal, and positional-aware representations across diverse domains, modalities, and tasks. We will release both the dataset and model implementations on GitHub.

Figures (10)

• Figure 1: Illustration of sequential multiple-instance learning for predicting artists' success with respect to a specific criterion. Given an artist's body of work as a sequence (depicted in red) of sets of artworks (depicted in cyan) created at discrete timesteps, the task is to predict the artist's success according to the criterion. The horizontal axis represents the artist’s career stage, while the vertical axis the temporal aspect. We utilize positional encodings to represent the relative order of discrete timesteps within an artist's career, with learnable embeddings for the absolute time values (e.g., year) associated with each timestep.
• Figure 2: This figure illustrates the Set2Seq Transformer. For an ordered sequence $\mathcal{T}$ of unordered sets $\mathcal{S}$, we learn a representation $\mathbf{s}_i$ for each set at timestep $t_i$. Positional encoding $\mathbf{u}_i$ and temporal embedding $\mathbf{v}_i$ are used for each timestep $t_i$, and summed with $\mathbf{s}_i$ to obtain the final representation $\boldsymbol{\tau}_i$ for the timestep $t_i$. These representations are then passed through a Transformer encoder, followed by a fully-connected layer, to generate the final prediction $\hat{y}$, derived from the last timestep $t_N$.
• Figure 3: Performance on early short-term wildfire danger forecasting using Mesogeos mesogeos. Given progressively earlier timesteps and limited temporal context, the task is to predict wildfire danger with incomplete information. PR-AUC scores are reported for various models and settings across different sequence lengths.
• Figure 5: Pairwise win-rate heatmap showing the fraction of times Model A (rows) defeats Model B (columns) in pairwise comparisons for the Stratified Split (left) and Time Series Split (right). Models are ordered by their average win-rate across all comparisons.
• Figure 6: Visualization of pairwise cosine distances between the first ten years of the top-2 ranked artists' careers based on the Aggregate Ranking (Pablo Picasso and Andy Warhol). (a) illustrates the pairwise cosine distance between the set representations of the first ten years of Pablo Picasso and Andy Warhol's careers. (b) illustrates the Position Encoding. (c) illustrates the Temporal Embeddings. (d) illustrates the aggregate of the set representations, positional encodings and temporal embeddings. (e) illustrates the final output obtained from the Transformer module. Lighter denotes higher cosine similarity.