TernaryCLIP: Efficiently Compressing Vision-Language Models with Ternary Weights and Distilled Knowledge

TL;DR

This work introduces TernaryCLIP, a framework that compresses both vision and text encoders of CLIP to ternary weights while preserving zero-shot cross-modal capabilities. It combines ternarization with three distillation strategies—contrastive relational distillation, interactive contrastive learning, and feature distillation—trained end-to-end using a ternarization-aware objective and a scaling mechanism that maps full-precision weights to a ternary format. Empirically, TernaryCLIP achieves up to 99% ternary weights with a 1.58-bit representation, 16.98× compression, 2.3× inference acceleration, and significant storage/memory reductions across 41 datasets, often outperforming 4-bit PTQ baselines in zero-shot classification and image-text retrieval. The results indicate that extreme quantization for large multimodal models is practical for edge and resource-constrained deployment, offering meaningful efficiency gains with controllable performance loss.

Abstract

Recent years have witnessed an increasing interest in image-text contrastive modeling, exemplified by models such as Contrastive Language-Image Pretraining (CLIP). In this paper, we propose the TernaryCLIP, a lightweight computational framework that converts connection weights of both vision and text encoders of CLIP into the ternary format, instead of full-precision or floating ones. TernaryCLIP incorporates quantization-aware training and distillation modules, preventing precision degradation and enabling low-cost and high-efficiency computations. Comprehensive experiments demonstrate that TernaryCLIP can achieve up to 99\% ternarized weights with 1.58-bit representation, 16.98 $\times$ compression ratio, 2.3 $\times$ inference acceleration, 16 $\times$ storage reduction, 10 $\times$ memory optimization, and 60\% sparsity while maintaining promising performance on zero-shot image classification and image-text retrieval tasks across 41 commonly used datasets. Our work highlights the feasibility of extreme quantization for large multimodal models, supporting effective and efficient deployment on resource-constrained devices. The model and code can be accessed from Hugging Face and GitHub.

Figures (18)

• Figure 1: The workflow of the TernaryCLIP framework, comprising two modules: ternarization and distillation.
• Figure 2: Zero-shot image classification performance (Accuracy@1) on three dataset types and CLIP models. ViT-L/14 or ViT-B/16 indicates which image encoder is used for the structure of CLIP.
• Figure 3: Zero-shot multi-label classification performance (mean average precision) on PASCAL VOC 2007. ViT-L/14 or ViT-B/16 indicates which image encoder is used for the structure of CLIP.
• Figure 4: Zero-shot image-to-text retrieval performance (Recall@5) on three datasets and CLIP models. ViT-L/14 or ViT-B/16 indicates which image encoder is used for the structure of CLIP.
• Figure 5: Zero-shot text-to-image retrieval performance (Recall@5) on three datasets and CLIP models. ViT-L/14 or ViT-B/16 indicates which image encoder is used for the structure of CLIP.