Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Winning Amazon KDD Cup'24

Chris Deotte, Ivan Sorokin, Ahmet Erdem, Benedikt Schifferer, Gilberto Titericz, Simon Jegou

TL;DR

This work presents the winning solution for the Amazon KDD Cup 2024 Multi Task Online Shopping Challenge, achieving first place across all five tracks. The approach centers on fine-tuning a single base model per track (Qwen2-72B-Instruct) on a richly constructed training set synthesized from real data and LLM-generated stimuli, followed by ensemble of multiple LoRA adapters and wise-ft to handle distribution shifts. Inference is accelerated via AWQ 4-bit quantization and vLLM, meeting strict per-track time limits while maintaining high accuracy and task compliance through logits processors. Ablation studies and comparisons with zero-shot baselines demonstrate the value of combined data curation, supervised fine-tuning, adapter ensembling, and constrained decoding in delivering robust, scalable shopping-assistant capabilities.

Abstract

This paper describes the winning solution of all 5 tasks for the Amazon KDD Cup 2024 Multi Task Online Shopping Challenge for LLMs. The challenge was to build a useful assistant, answering questions in the domain of online shopping. The competition contained 57 diverse tasks, covering 5 different task types (e.g. multiple choice) and across 4 different tracks (e.g. multi-lingual). Our solution is a single model per track. We fine-tune Qwen2-72B-Instruct on our own training dataset. As the competition released only 96 example questions, we developed our own training dataset by processing multiple public datasets or using Large Language Models for data augmentation and synthetic data generation. We apply wise-ft to account for distribution shifts and ensemble multiple LoRA adapters in one model. We employed Logits Processors to constrain the model output on relevant tokens for the tasks. AWQ 4-bit Quantization and vLLM are used during inference to predict the test dataset in the time constraints of 20 to 140 minutes depending on the track. Our solution achieved the first place in each individual track and is the first place overall of Amazons KDD Cup 2024.

Winning Amazon KDD Cup'24

TL;DR

This work presents the winning solution for the Amazon KDD Cup 2024 Multi Task Online Shopping Challenge, achieving first place across all five tracks. The approach centers on fine-tuning a single base model per track (Qwen2-72B-Instruct) on a richly constructed training set synthesized from real data and LLM-generated stimuli, followed by ensemble of multiple LoRA adapters and wise-ft to handle distribution shifts. Inference is accelerated via AWQ 4-bit quantization and vLLM, meeting strict per-track time limits while maintaining high accuracy and task compliance through logits processors. Ablation studies and comparisons with zero-shot baselines demonstrate the value of combined data curation, supervised fine-tuning, adapter ensembling, and constrained decoding in delivering robust, scalable shopping-assistant capabilities.

Abstract

This paper describes the winning solution of all 5 tasks for the Amazon KDD Cup 2024 Multi Task Online Shopping Challenge for LLMs. The challenge was to build a useful assistant, answering questions in the domain of online shopping. The competition contained 57 diverse tasks, covering 5 different task types (e.g. multiple choice) and across 4 different tracks (e.g. multi-lingual). Our solution is a single model per track. We fine-tune Qwen2-72B-Instruct on our own training dataset. As the competition released only 96 example questions, we developed our own training dataset by processing multiple public datasets or using Large Language Models for data augmentation and synthetic data generation. We apply wise-ft to account for distribution shifts and ensemble multiple LoRA adapters in one model. We employed Logits Processors to constrain the model output on relevant tokens for the tasks. AWQ 4-bit Quantization and vLLM are used during inference to predict the test dataset in the time constraints of 20 to 140 minutes depending on the track. Our solution achieved the first place in each individual track and is the first place overall of Amazons KDD Cup 2024.
Paper Structure (15 sections, 2 equations, 3 figures, 7 tables)

This paper contains 15 sections, 2 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Amazon KDDCup 2024: Multi Task Online Shopping Challenge for LLMs
  3. Training Dataset
  4. Real Datasets
  5. Synthetic Datasets
  6. Model
  7. Prompt Template
  8. Fine-Tuning Qwen2
  9. Ensemble Adapters
  10. Wise-ft
  11. Logits Processors
  12. Quantization / vLLM
  13. Results
  14. Conclusion
  15. Details on Training Dataset

Figures (3)

  • Figure 1: One example of the development dataset. It is a multiple choice question answering tasks for understanding shopping concepts.
  • Figure 2: High-Level Oerview of our pipeline for the KDD Cup 2024
  • Figure 3: Accuracy gain on Track 5 using various $\alpha$ for wise-ft for 3 different versions of LoRA adapters (v7, v8 and the ensemble v7 + v8)