Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models

Zhen Yang, Yingxue Zhang, Fandong Meng, Jie Zhou

TL;DR

The paper tackles the challenge of efficiently enabling multi-modal large language models to understand and generate across text, image, and audio without extensive retraining or multiple tools. TEAL introduces a token-in-token-out architecture that tokenizes all modalities into a shared token sequence and learns a joint embedding space, allowing a frozen textual LLM to perform multimodal tasks with minimal parameter updates via a two-stage supervised finetuning process. Empirical results show TEAL improves multimodal understanding on COCO-Caption, ScienceQA, and CoVoST 2, and enables simple text-to-image generation, with tokenizer choice and vocabulary size emerging as critical factors. The work suggests a promising path toward unified multimodal processing and generation, highlighting the importance of semantic-rich tokenizers and efficient fine-tuning strategies for broad modality coverage.

Abstract

Despite Multi-modal Large Language Models (MM-LLMs) have made exciting strides recently, they are still struggling to efficiently model the interactions among multi-modal inputs and the generation in non-textual modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an approach to treat the input from any modality as a token sequence and learn a joint embedding space for all modalities. Specifically, for the input from any modality, TEAL first discretizes it into a token sequence with the off-the-shelf tokenizer and embeds the token sequence into a joint embedding space with a learnable embedding matrix. MM-LLMs just need to predict the multi-modal tokens autoregressively as the textual LLMs do. Finally, the corresponding de-tokenizer is applied to generate the output in each modality based on the predicted token sequence. With the joint embedding space, TEAL enables the frozen LLMs to perform both understanding and generation tasks involving non-textual modalities, such as image and audio. Thus, the textual LLM can just work as an interface and maintain its high performance in textual understanding and generation. Experiments show that TEAL achieves substantial improvements in multi-modal understanding, and implements a simple scheme for multi-modal generations.

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models

TL;DR

The paper tackles the challenge of efficiently enabling multi-modal large language models to understand and generate across text, image, and audio without extensive retraining or multiple tools. TEAL introduces a token-in-token-out architecture that tokenizes all modalities into a shared token sequence and learns a joint embedding space, allowing a frozen textual LLM to perform multimodal tasks with minimal parameter updates via a two-stage supervised finetuning process. Empirical results show TEAL improves multimodal understanding on COCO-Caption, ScienceQA, and CoVoST 2, and enables simple text-to-image generation, with tokenizer choice and vocabulary size emerging as critical factors. The work suggests a promising path toward unified multimodal processing and generation, highlighting the importance of semantic-rich tokenizers and efficient fine-tuning strategies for broad modality coverage.

Abstract

Despite Multi-modal Large Language Models (MM-LLMs) have made exciting strides recently, they are still struggling to efficiently model the interactions among multi-modal inputs and the generation in non-textual modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an approach to treat the input from any modality as a token sequence and learn a joint embedding space for all modalities. Specifically, for the input from any modality, TEAL first discretizes it into a token sequence with the off-the-shelf tokenizer and embeds the token sequence into a joint embedding space with a learnable embedding matrix. MM-LLMs just need to predict the multi-modal tokens autoregressively as the textual LLMs do. Finally, the corresponding de-tokenizer is applied to generate the output in each modality based on the predicted token sequence. With the joint embedding space, TEAL enables the frozen LLMs to perform both understanding and generation tasks involving non-textual modalities, such as image and audio. Thus, the textual LLM can just work as an interface and maintain its high performance in textual understanding and generation. Experiments show that TEAL achieves substantial improvements in multi-modal understanding, and implements a simple scheme for multi-modal generations.
Paper Structure (23 sections, 2 equations, 3 figures, 8 tables)

This paper contains 23 sections, 2 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MM-LLMs
  4. Non-textual discretization
  5. VQ-VAEs
  6. Clustering
  7. Method
  8. Model Architecture
  9. Tokenize and De-Tokenize
  10. Two-stage Supervised Finetuning
  11. Pre-training
  12. Fine-tuning
  13. Experiments
  14. COCO-Caption
  15. ScienceQA
  16. ...and 8 more sections

Figures (3)

  • Figure 1: The main architecture of TEAL. The modules in MM-LLM denoted with the color gray make up the original textual LLM and most of them are frozen during training.
  • Figure 2: Some examples in the coco2014 validation set. For each case, we present the original image ID, the prompt, the output of our model, and one reference caption randomly selected among all five references.
  • Figure 3: Some examples of the text-to-image generation on MNIST test set. We test with both simple and complex questions for the proposed TEAL.