One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code

TL;DR

SkillNet tackles the problem of multitask multimodal learning with a single sparsely activated Transformer. It introduces modality-specific skills that gate Q/K/V projections, enabling selective activation across text, image, sound, video, and code. Across downstream and pretraining tasks, SkillNet achieves performance comparable to modality-specific baselines and gains from sparse pretraining, with notable gains in efficiency for Chinese text-to-image retrieval. This approach offers a scalable, interpretable path toward unified multimodal understanding that can extend to more modalities and tasks in the future.

Abstract

People perceive the world with multiple senses (e.g., through hearing sounds, reading words and seeing objects). However, most existing AI systems only process an individual modality. This paper presents an approach that excels at handling multiple modalities of information with a single model. In our "{SkillNet}" model, different parts of the parameters are specialized for processing different modalities. Unlike traditional dense models that always activate all the model parameters, our model sparsely activates parts of the parameters whose skills are relevant to the task. Such model design enables SkillNet to learn skills in a more interpretable way. We develop our model for five modalities including text, image, sound, video and code. Results show that, SkillNet performs comparably to five modality-specific fine-tuned models. Moreover, our model supports self-supervised pretraining with the same sparsely activated way, resulting in better initialized parameters for different modalities. We find that pretraining significantly improves the performance of SkillNet on five modalities, on par with or even better than baselines with modality-specific pretraining. On the task of Chinese text-to-image retrieval, our final system achieves higher accuracy than existing leading systems including Wukong{ViT-B} and Wenlan 2.0 while using less number of activated parameters.

Figures (8)

• Figure 1: In SkillNet, each pillar refers to a skill. Pillars filled in color (e.g., yellow, blue, green, purple and red) are activated.
• Figure 2: An illustration of image search with Transformer-based Siamese network.
• Figure 3: Architecture of SkillNet for image retrieval. Text encoder and image encoder are two pathways of one shared model — s$_{text}$ and s$_{image}$ are activated for the text encoder and the image encoder, respectively.
• Figure 4: An illustration of the pipeline and the embeddings of different modalities.
• Figure 5: Performance of SkillNet with different finetuning steps. X-axis stands for the training steps. Y-axis stands for the evaluation metric (lower is better for CER).