Fine-tuning Multimodal Large Language Models for Product Bundling

TL;DR

This work targets product bundling by leveraging multimodal information (text, visual, acoustic, and relational signals) through a fine-tuned multimodal large language model (MLLM) framework, Bundle-MLLM. It introduces hybrid item tokenization to embed textual and non-textual item features into a single token, and uses a soft separator to distinguish modalities, followed by a lightweight fusion module to align with LLM representations. Bundling tasks are reformulated as a multiple-choice problem, and a progressive optimization strategy (text-only bundle pattern learning via LoRA, then multimodal semantic alignment) enables effective adaptation of LLMs to bundling knowledge. Across four datasets in two domains, Bundle-MLLM outperforms conventional methods and strong LLM baselines, with high validity in responses and improved inference efficiency, demonstrating robust multimodal understanding and practical deployment potential. The work also shows strong performance in cold-start settings and scalable improvements with larger LLM backbones, pointing toward future personalization and handling longer candidate sets.

Abstract

Recent advances in product bundling have leveraged multimodal information through sophisticated encoders, but remain constrained by limited semantic understanding and a narrow scope of knowledge. Therefore, some attempts employ In-context Learning (ICL) to explore the potential of large language models (LLMs) for their extensive knowledge and complex reasoning abilities. However, these efforts are inadequate in understanding mulitmodal data and exploiting LLMs' knowledge for product bundling. To bridge the gap, we introduce Bundle-MLLM, a novel framework that fine-tunes LLMs through a hybrid item tokenization approach within a well-designed optimization strategy. Specifically, we integrate textual, media, and relational data into a unified tokenization, introducing a soft separation token to distinguish between textual and non-textual tokens. Additionally, a streamlined yet powerful multimodal fusion module is employed to embed all non-textual features into a single, informative token, significantly boosting efficiency. To tailor product bundling tasks for LLMs, we reformulate the task as a multiple-choice question with candidate items as options. We further propose a progressive optimization strategy that fine-tunes LLMs for disentangled objectives: 1) learning bundle patterns and 2) enhancing multimodal semantic understanding specific to product bundling. Extensive experiments on four datasets across two domains demonstrate that our approach outperforms a range of state-of-the-art (SOTA) methods.

Figures (8)

• Figure 1: Illustrative example of how the multiple semantics from intrinsically multimodal data and extensive knowledge benefit the product bundling across different domains (e.g., outfits or playlists).
• Figure 2: The overall framework of Bundle-MLLM, which incorporates multiple data formats, including textual, visual/acoustic, and relational features (i.e., user-item interactions and bundle-item affiliations). These heterogeneous features are extracted from various foundation encoders and embedded into a single multimodal token via a trainable fusion module, followed by a projector to align with the LLM space.
• Figure 3: The performance w.r.t. different optimization stages.
• Figure 4: The performance w.r.t. different numbers of candidate items.
• Figure 5: The training loss curve across different optimization stages.