Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis

TL;DR

This paper tackles the suboptimal class separation and high dimensionality of raw CLIP embeddings for supervised classification. It introduces Koo-Fu CLIP, a closed-form adaptation that applies the Fukunaga–Koontz Transform in a whitened embedding space to simultaneously suppress within-class variation and boost between-class discrimination, producing a low-dimensional discriminative subspace. Empirically, this approach improves nearest-prototype accuracy on ImageNet from 75.1% to 79.1% (IN-1K) and remains robust as the label space expands to 14K and 21K, while enabling 10–12× dimensionality reduction with minimal accuracy loss. The method is lightweight, requires only a single regularization parameter, and opens practical avenues for scalable large-scale classification and retrieval with vision–language models, motivating further theoretical analysis and extension to other multimodal architectures.

Abstract

Visual-language models such as CLIP provide powerful general-purpose representations, but their raw embeddings are not optimized for supervised classification, often exhibiting limited class separation and excessive dimensionality. We propose Koo-Fu CLIP, a supervised CLIP adaptation method based on Fukunaga-Koontz Linear Discriminant Analysis, which operates in a whitened embedding space to suppress within-class variation and enhance between-class discrimination. The resulting closed-form linear projection reshapes the geometry of CLIP embeddings, improving class separability while performing effective dimensionality reduction, and provides a lightweight and efficient adaptation of CLIP representations. Across large-scale ImageNet benchmarks, nearest visual prototype classification in the Koo-Fu CLIP space improves top-1 accuracy from 75.1% to 79.1% on ImageNet-1K, with consistent gains persisting as the label space expands to 14K and 21K classes. The method supports substantial compression by up to 10-12x with little or no loss in accuracy, enabling efficient large-scale classification and retrieval.