Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dense Communication between Language Models

Shiguang Wu, Yaqing Wang, Quanming Yao

TL;DR

This paper proposes a novel paradigm of direct dense vector communication between Large Language Models, eliminating the unnecessary embedding and de-embedding steps when LLM interact with another, enabling more efficient information transfer, fully differentiable optimization pathways, and exploration of capabilities beyond human heuristics.

Abstract

As higher-level intelligence emerges from the combination of modular components with lower-level intelligence, many works combines Large Language Models (LLMs) for collective intelligence. Such combination is achieved by building communications among LLMs. While current systems primarily facilitate such communication through natural language, this paper proposes a novel paradigm of direct dense vector communication between LLMs. Our approach eliminates the unnecessary embedding and de-embedding steps when LLM interact with another, enabling more efficient information transfer, fully differentiable optimization pathways, and exploration of capabilities beyond human heuristics. We use such stripped LLMs as vertexes and optimizable seq2seq modules as edges to construct LMNet, with similar structure as MLPs. By utilizing smaller pre-trained LLMs as vertexes, we train a LMNet that achieves comparable performance with LLMs in similar size with only less than 0.1% training cost. This offers a new perspective on scaling for general intelligence rather than training a monolithic LLM from scratch. Besides, the proposed method can be used for other applications, like customizing LLM with limited data, showing its versatility.

Dense Communication between Language Models

TL;DR

This paper proposes a novel paradigm of direct dense vector communication between Large Language Models, eliminating the unnecessary embedding and de-embedding steps when LLM interact with another, enabling more efficient information transfer, fully differentiable optimization pathways, and exploration of capabilities beyond human heuristics.

Abstract

As higher-level intelligence emerges from the combination of modular components with lower-level intelligence, many works combines Large Language Models (LLMs) for collective intelligence. Such combination is achieved by building communications among LLMs. While current systems primarily facilitate such communication through natural language, this paper proposes a novel paradigm of direct dense vector communication between LLMs. Our approach eliminates the unnecessary embedding and de-embedding steps when LLM interact with another, enabling more efficient information transfer, fully differentiable optimization pathways, and exploration of capabilities beyond human heuristics. We use such stripped LLMs as vertexes and optimizable seq2seq modules as edges to construct LMNet, with similar structure as MLPs. By utilizing smaller pre-trained LLMs as vertexes, we train a LMNet that achieves comparable performance with LLMs in similar size with only less than 0.1% training cost. This offers a new perspective on scaling for general intelligence rather than training a monolithic LLM from scratch. Besides, the proposed method can be used for other applications, like customizing LLM with limited data, showing its versatility.
Paper Structure (20 sections, 1 theorem, 2 equations, 4 figures, 4 tables)

This paper contains 20 sections, 1 theorem, 2 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Stripping Embedding Layers
  5. Constructing LMNet
  6. Training
  7. Discussion
  8. Specification of Modules
  9. Topology Hypothesis
  10. Complexity Analysis
  11. Collective Intelligence in LMNet
  12. Applications and Empirical Results
  13. Scaling for General Intelligence
  14. Performance of (Re)Pre-Trained LMNet
  15. A Closer Look at How LMNet Makes Inference through Dense Communication
  16. ...and 5 more sections

Key Result

Proposition 1

Given a differentiable supervision signal $\mathcal{L}$, i.e.,$\frac{\partial \mathcal{L}}{\partial \bm{p}^o}$, we can obtain gradient on all parameters in LMNet, i.e., $\frac{\partial \mathcal{L}}{\partial \theta^l_i}$ and $\frac{\partial \mathcal{L}}{\partial \mathbf{\omega}^l_{ij}}$ for any $l,i,

Figures (4)

  • Figure 1: Communication between LLMs through dense vectors eliminates the bottleneck of natural language.
  • Figure 2: Illustration of the proposed paradigm. (a) A standard LLM processes discrete token inputs by embedding them into dense vectors, and outputs discrete tokens via a de-embedding layer. (b) Existing communication between LLMs typically occurs through discrete tokens. (c) Our approach strips the embedding and de-embedding layers, allowing LLMs to communicate directly via dense vectors. (d) We construct and train a LMNet by connecting stripped transformers with trainable communication modules.
  • Figure 3: Visualization of attention weights in the edge modules on the 4 edges at the last layer of trained LMNet-1B, given certain input sentence (first 100 tokens only).
  • Figure 4: Visualization of query projection matrix of the attention block on every edge in trained LMNet-1B. All edges are shown under the same value-color mapping.

Theorems & Definitions (1)

  • Proposition 1: End-to-End Gradient Descent