Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preference Packing: Efficient Preference Optimization for Large Language Models

Jaekyung Cho

TL;DR

Preference packing is proposed, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO), resulting in a 3.22x speedup.

Abstract

Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preference packing, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO). Preference packing improves resource efficiency by reducing the attention operations for duplicate input prompts and decreasing KV cache memory usage. We conducted experiments on text-only datasets and image-included datasets and achieved at least 37% reduction in training time. Notably, this method can be applied alongside existing optimization techniques such as batch sorting, resulting in a 3.22x speedup.

Preference Packing: Efficient Preference Optimization for Large Language Models

TL;DR

Preference packing is proposed, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO), resulting in a 3.22x speedup.

Abstract

Resource-efficient training optimization techniques are becoming increasingly important as the size of large language models (LLMs) continues to grow. In particular, batch packing is commonly used in pre-training and supervised fine-tuning to achieve resource-efficient training. We propose preference packing, a method to enhance resource efficiency in training techniques that use data with different responses for the same input prompt, such as reward models or Direct Preference Optimization (DPO). Preference packing improves resource efficiency by reducing the attention operations for duplicate input prompts and decreasing KV cache memory usage. We conducted experiments on text-only datasets and image-included datasets and achieved at least 37% reduction in training time. Notably, this method can be applied alongside existing optimization techniques such as batch sorting, resulting in a 3.22x speedup.
Paper Structure (14 sections, 9 equations, 1 figure, 2 tables)

This paper contains 14 sections, 9 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preference optimization
  4. Training resource efficiency
  5. Methods
  6. Symbols
  7. Preference packing
  8. Theoretical performance
  9. Experimental Setups
  10. Training datasets
  11. Training Models
  12. Results and Analysis
  13. Resource efficiency
  14. Scalability

Figures (1)

  • Figure 1: Difference in data sequence format, position IDs, and attention masks. (a) By default, a single preference data sample is processed in a batch-wise manner, where input prompt unnecessarily repeated. (b) In preference packing, multiple preference responses are concatenated in a sequence-wise manner into a single sequence following the shared input prompt. Position IDs and attention masks are modified accordingly to reflect the new structure.