Integrating Domain Knowledge into Large Language Models for Enhanced Fashion Recommendations

TL;DR

The work addresses the challenge of providing personalized fashion recommendations that remain robust under distribution shifts by integrating domain knowledge into a large language model. It introduces the Fashion Large Language Model (FLLM) trained with auto-prompt generation and enhanced by Retrieval-Augmented Generation (RAG) to tailor suggestions to individual users. Key contributions include a domain-aware fine-tuning pipeline using FIIB and binary/FITB data, template QA and LLM auto QA prompts, and a multi-path retrieval mechanism that conditions recommendations on user context and fashion knowledge. Results demonstrate improved accuracy, strong few-shot learning, and better adaptability compared with baselines, underscoring the practical potential of combining LLMs with domain retrieval for personalized, explainable fashion guidance.

Abstract

Fashion, deeply rooted in sociocultural dynamics, evolves as individuals emulate styles popularized by influencers and iconic figures. In the quest to replicate such refined tastes using artificial intelligence, traditional fashion ensemble methods have primarily used supervised learning to imitate the decisions of style icons, which falter when faced with distribution shifts, leading to style replication discrepancies triggered by slight variations in input. Meanwhile, large language models (LLMs) have become prominent across various sectors, recognized for their user-friendly interfaces, strong conversational skills, and advanced reasoning capabilities. To address these challenges, we introduce the Fashion Large Language Model (FLLM), which employs auto-prompt generation training strategies to enhance its capacity for delivering personalized fashion advice while retaining essential domain knowledge. Additionally, by integrating a retrieval augmentation technique during inference, the model can better adjust to individual preferences. Our results show that this approach surpasses existing models in accuracy, interpretability, and few-shot learning capabilities.

Figures (5)

• Figure 1: Fashion item recommendations can differ significantly based on various occasions, styles, and attributes. (a) Traditional methods (red box), trained solely on fixed datasets using sequence models, are constrained to aligning with the training data and struggle to adapt to new preference distributions. In contrast, our (b) LLM methods (green box) dynamically integrate diverse contexts into the prompt. Leveraging the inference capabilities of large models, our methods produce results under a wide range of conditions.
• Figure 2: An overview of an information-augmented retrieval system featuring a fine-tuned Large Language Model (Fashion-LLM) is presented. This system integrates multiple inputs into the Fashion-LLM to generate precise product descriptions.
• Figure 3: Detailed Workflow of Model Fine-Tuning and Retrieval Processes for Fashion Recommender System. In this AI-powered system, we employ both template-based AQ generation and LLM auto-question generation to create domain-relevant questions and prompts.
• Figure 4: Performance of different methods with respect to FIIB at different of ratio of training data
• Figure 5: Visualization of recommendation results: For each query item, our model can generate different outfits based on various style preferences.