Ternarization of Vision Language Models for use on edge devices

TL;DR

It is found that the ternary model using the custom ternary matrix multiplication operator provides a good compromise in term of memory usage and perplexity, while having the fastest token generation speed.

Abstract

We propose a process to compress a pre-trained Vision Language Model into a ternary version of itself instead of training a ternary model from scratch. A new initialization scheme from pre-trained weights based on the k-means algorithm is proposed to reduce the ternarization time. We implement different custom operators for executing the ternary model on the TensorFlow Lite Engine. We compare the original model with its ternary and binary versions in terms of memory consumption, inference speed and perplexity. We find that the ternary model using our custom ternary matrix multiplication operator provides a good compromise in term of memory usage and perplexity, while having the fastest token generation speed.

Figures (4)

• Figure 1: Details of the quantization process of the LLM Transformer.
• Figure 2: Initial loss and quantization error by iteration of Algorithm \ref{['alg:kmeans_ternary']}
• Figure 3: Perplexity by file size (GB)
• Figure 4: Perplexity by model memory usage (GB)