Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Personalised Outfit Recommendation via History-aware Transformers

Myong Chol Jung, Julien Monteil, Philip Schulz, Volodymyr Vaskovych

TL;DR

This work tackles personalised outfit recommendation by leveraging shoppers' purchase history. It introduces the History-aware Transformer (HAT), a two-level transformer that encodes individual outfits and conditions their compatibility on a shopper's history, trained with a focal loss, supervised contrastive loss, and an adaptive margin loss that uses weak negatives. Empirical results on IQON3000 and Polyvore show significant improvements in outfit compatibility prediction (CP) and fill-in-the-blank (FITB) tasks, with ablations highlighting the contributions of history-based contrastive learning, weak negatives, and adaptive margins. The approach offers a practical pathway to more tailored fashion recommendations, while future work could extend retrieval capabilities, account for history recency, and integrate richer item metadata.

Abstract

We present the history-aware transformer (HAT), a transformer-based model that uses shoppers' purchase history to personalise outfit predictions. The aim of this work is to recommend outfits that are internally coherent while matching an individual shopper's style and taste. To achieve this, we stack two transformer models, one that produces outfit representations and another one that processes the history of purchased outfits for a given shopper. We use these models to score an outfit's compatibility in the context of a shopper's preferences as inferred from their previous purchases. During training, the model learns to discriminate between purchased and random outfits using 3 losses: the focal loss for outfit compatibility typically used in the literature, a contrastive loss to bring closer learned outfit embeddings from a shopper's history, and an adaptive margin loss to facilitate learning from weak negatives. Together, these losses enable the model to make personalised recommendations based on a shopper's purchase history. Our experiments on the IQON3000 and Polyvore datasets show that HAT outperforms strong baselines on the outfit Compatibility Prediction (CP) and the Fill In The Blank (FITB) tasks. The model improves AUC for the CP hard task by 15.7% (IQON3000) and 19.4% (Polyvore) compared to previous SOTA results. It further improves accuracy on the FITB hard task by 6.5% and 9.7%, respectively. We provide ablation studies on the personalisation, constrastive loss, and adaptive margin loss that highlight the importance of these modelling choices.

Personalised Outfit Recommendation via History-aware Transformers

TL;DR

This work tackles personalised outfit recommendation by leveraging shoppers' purchase history. It introduces the History-aware Transformer (HAT), a two-level transformer that encodes individual outfits and conditions their compatibility on a shopper's history, trained with a focal loss, supervised contrastive loss, and an adaptive margin loss that uses weak negatives. Empirical results on IQON3000 and Polyvore show significant improvements in outfit compatibility prediction (CP) and fill-in-the-blank (FITB) tasks, with ablations highlighting the contributions of history-based contrastive learning, weak negatives, and adaptive margins. The approach offers a practical pathway to more tailored fashion recommendations, while future work could extend retrieval capabilities, account for history recency, and integrate richer item metadata.

Abstract

We present the history-aware transformer (HAT), a transformer-based model that uses shoppers' purchase history to personalise outfit predictions. The aim of this work is to recommend outfits that are internally coherent while matching an individual shopper's style and taste. To achieve this, we stack two transformer models, one that produces outfit representations and another one that processes the history of purchased outfits for a given shopper. We use these models to score an outfit's compatibility in the context of a shopper's preferences as inferred from their previous purchases. During training, the model learns to discriminate between purchased and random outfits using 3 losses: the focal loss for outfit compatibility typically used in the literature, a contrastive loss to bring closer learned outfit embeddings from a shopper's history, and an adaptive margin loss to facilitate learning from weak negatives. Together, these losses enable the model to make personalised recommendations based on a shopper's purchase history. Our experiments on the IQON3000 and Polyvore datasets show that HAT outperforms strong baselines on the outfit Compatibility Prediction (CP) and the Fill In The Blank (FITB) tasks. The model improves AUC for the CP hard task by 15.7% (IQON3000) and 19.4% (Polyvore) compared to previous SOTA results. It further improves accuracy on the FITB hard task by 6.5% and 9.7%, respectively. We provide ablation studies on the personalisation, constrastive loss, and adaptive margin loss that highlight the importance of these modelling choices.
Paper Structure (31 sections, 7 equations, 5 figures, 6 tables)

This paper contains 31 sections, 7 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Outfit Recommendation
  4. Non-personalised outfits
  5. Personalised outfits
  6. Sequential Recommendation
  7. Contrastive Learning
  8. Methodology
  9. Problem Formulation
  10. History-aware Transformer (HAT)
  11. Bottom-level Encoder
  12. Top-level Encoder
  13. Model Training
  14. Outfit Compatibility
  15. Personalisation by Contrastive Learning
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Model diagram of HAT. The target outfit is an outfit whose compatibility score we wish to compute. It consists of images and titles for each item. K, V, and Q represent key, value, and query respectively for the cross-attention. The bottom-level transformer encoder shares weights.
  • Figure 2: Exemplary outfit embeddings projected to 2D space by t-SNE van2008visualizing in IQON3000. Colours indicate different shoppers.
  • Figure 3: Examples of positive outfit (a), negative outfit (b), and weak negative outfit (c). Every item of the negative outfit is randomly selected item in place of the positive items within the same item category. On the other hand, a weak negative outfit only has a single item switched.
  • Figure 4: Illustration of the adaptive margin loss. Score (w) indicates the compatibility score of a weak negative outfit, and Score (p) indicates the compatibility score of a positive outfit. The margin is weighted by the learned attention weight of the switched item representing its importance in determining the compatibility of an outfit.
  • Figure 5: An example of a pair of positive outfit and negative outfit in CP-Hard. Shopper A and Shopper B have different positive outfits that share the same watch. For Shopper A, we consider the positive outfit of Shopper B as a negative outfit, and for Shopper B, we consider the positive outfit of Shopper A as a negative outfit.