Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Models

Wei Chen, Zhiyuan Li, Shuo Xin, Yihao Wang

TL;DR

This work tackles the challenge of energy-efficient on-device language modeling with long contexts. It introduces Squid, a decoder-decoder architecture that compresses extended context into memory tokens using a small 0.5B decoder, which are then processed by a 7B main decoder via a vision-language-style projector, effectively treating long text as a separate modality. The approach is trained in three stages—restoration, continual, and instruction fine-tuning—alongside memory-token mechanisms to preserve semantic fidelity. Empirically, Squid delivers up to 10× energy efficiency and 5× latency reduction while maintaining high task accuracy, enabling more capable on-device NLP in resource-constrained environments and broadening edge-AI applicability.

Abstract

This paper presents Dolphin, a novel decoder-decoder architecture for energy-efficient processing of long contexts in language models. Our approach addresses the significant energy consumption and latency challenges inherent in on-device models. Dolphin employs a compact 0.5B parameter decoder to distill extensive contextual information into a memory embedding, substantially reducing the input length for the primary 7B parameter decoder model. Inspired by vision-language models, we repurpose the image embedding projector to encode long textual contexts, effectively treating extended context as a distinct modality. This innovative method enables processing of substantially longer contexts without the typical computational overhead associated with extended input sequences. Empirical evaluations demonstrate a 10-fold improvement in energy efficiency and a 5-fold reduction in latency compared to conventional full-length context processing methods without losing quality of the response. Our work contributes to the development of more sustainable and scalable language models for on-device applications, addressing the critical need for energy-efficient and responsive AI technologies in resource-constrained environments while maintaining the accuracy to understand long contexts. This research has implications for the broader field of natural language processing, particularly in the domain of efficient model design for resource-limited settings. By enabling more sophisticated AI capabilities on edge devices, Dolphin paves the way for advanced language processing in a wide range of applications where computational resources are at a premium. The Dolphin model is publicly available at https://huggingface.co/NexaAIDev/Dolphin.

Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Models

TL;DR

This work tackles the challenge of energy-efficient on-device language modeling with long contexts. It introduces Squid, a decoder-decoder architecture that compresses extended context into memory tokens using a small 0.5B decoder, which are then processed by a 7B main decoder via a vision-language-style projector, effectively treating long text as a separate modality. The approach is trained in three stages—restoration, continual, and instruction fine-tuning—alongside memory-token mechanisms to preserve semantic fidelity. Empirically, Squid delivers up to 10× energy efficiency and 5× latency reduction while maintaining high task accuracy, enabling more capable on-device NLP in resource-constrained environments and broadening edge-AI applicability.

Abstract

This paper presents Dolphin, a novel decoder-decoder architecture for energy-efficient processing of long contexts in language models. Our approach addresses the significant energy consumption and latency challenges inherent in on-device models. Dolphin employs a compact 0.5B parameter decoder to distill extensive contextual information into a memory embedding, substantially reducing the input length for the primary 7B parameter decoder model. Inspired by vision-language models, we repurpose the image embedding projector to encode long textual contexts, effectively treating extended context as a distinct modality. This innovative method enables processing of substantially longer contexts without the typical computational overhead associated with extended input sequences. Empirical evaluations demonstrate a 10-fold improvement in energy efficiency and a 5-fold reduction in latency compared to conventional full-length context processing methods without losing quality of the response. Our work contributes to the development of more sustainable and scalable language models for on-device applications, addressing the critical need for energy-efficient and responsive AI technologies in resource-constrained environments while maintaining the accuracy to understand long contexts. This research has implications for the broader field of natural language processing, particularly in the domain of efficient model design for resource-limited settings. By enabling more sophisticated AI capabilities on edge devices, Dolphin paves the way for advanced language processing in a wide range of applications where computational resources are at a premium. The Dolphin model is publicly available at https://huggingface.co/NexaAIDev/Dolphin.
Paper Structure (16 sections, 9 equations, 1 figure, 6 tables)

This paper contains 16 sections, 9 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Long Context as a Novel Modality
  5. Memory Tokens
  6. Multi-stage Training
  7. Restoration Training
  8. Continual Training
  9. Instruction Fine-tuning
  10. Comparison with Vision-Language Model Training
  11. Dataset
  12. Experiments
  13. Testing Datasets
  14. Restoration performance
  15. Compression Performance
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The model architecture of the Squid model, with three different components, the text encoder which is a model with transformer decoder architecture. The projector is to convert the embedding information after text encoder into the embedding that can be understood by the main LLM, which is another transformers decoder model.