Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoDification: Mixture of Depths Made Easy

Chen Zhang, Meizhi Zhong, Qimeng Wang, Xuantao Lu, Zheyu Ye, Chengqiang Lu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang, Dawei Song

TL;DR

It is discovered that MoD can barely transform existing LLMs without costly training over an extensive number of tokens, so to enable the transformations from any LLMs to MoD ones, top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should be crafted along.

Abstract

Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications.

MoDification: Mixture of Depths Made Easy

TL;DR

It is discovered that MoD can barely transform existing LLMs without costly training over an extensive number of tokens, so to enable the transformations from any LLMs to MoD ones, top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should be crafted along.

Abstract

Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications.

Paper Structure

This paper contains 19 sections, 5 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Long-context Efficiency
  4. Mixture of Depths
  5. MoDification
  6. Architecture
  7. Data
  8. Experiments
  9. Setup
  10. Implementation
  11. Results
  12. Ablation Studies
  13. Execution Visualization
  14. Latency Profiling
  15. Long-context Applications
  16. ...and 4 more sections

Figures (9)

  • Figure 1: The sub-optimality of MoD due to the use of top-k operator. For efficiency, the efficiency improvement is restricted because top-k operator is not cheap and unimportant computations are preserved by retaining a constant number of tokens at every layer. For effectiveness, the effectiveness is undesired because dominated computations are saved by skipping a constant number of tokens at every layer.
  • Figure 2: The comparison among transformer, early exiting (EE), mixture of depths (MoD), and our MoDification.
  • Figure 3: The training losses. The steady loss curves indicate that both MoD and MoDification are rigorously optimized.
  • Figure 4: The comparison results concerning other efficient designs. ShortGPT is more efficient yet less effective.
  • Figure 5: The impact of threshold value $p$.
  • ...and 4 more figures