KCM: KAN-Based Collaboration Models Enhance Pretrained Large Models

TL;DR

This work introduces KCM, a KAN-based collaboration framework that routes inputs between a small KAN-based model and pretrained large models to reduce compute while preserving accuracy. By employing a confidence-based routing threshold $\epsilon$ on the small-model score $C_x$ and leveraging bidirectional synergy (prompt modification for the large model and knowledge distillation back to the small model via KL divergence when $C_l > \epsilon$), KCM achieves substantial reductions in large-model invocations across language, vision, and multimodal tasks. The approach demonstrates improved tail-data performance due to KAN’s edge-based, learnable activations that mitigate catastrophic forgetting, and consistently outperforms MLP-based collaboration models (MCM) in both efficiency and accuracy. Overall, KCM offers a practical, generalizable pathway to deploy PLMs more cost-effectively without compromising task performance, particularly in long-tail and cross-modal settings.

Abstract

In recent years, Pretrained Large Models(PLMs) researchers proposed large-small model collaboration frameworks, leveraged easily trainable small models to assist large models, aim to(1) significantly reduce computational resource consumption while maintaining comparable accuracy, and (2) enhance large model performance in specialized domain tasks. However, this collaborative paradigm suffers from issues such as significant accuracy degradation, exacerbated catastrophic forgetting, and amplified hallucination problems induced by small model knowledge. To address these challenges, we propose a KAN-based Collaborative Model (KCM) as an improved approach to large-small model collaboration. The KAN utilized in KCM represents an alternative neural network architecture distinct from conventional MLPs. Compared to MLPs, KAN offers superior visualizability and interpretability while mitigating catastrophic forgetting. We deployed KCM in large-small model collaborative systems across three scenarios: language, vision, and vision-language cross-modal tasks. The experimental results demonstrate that, compared with pure large model approaches, the large-small model collaboration framework utilizing KCM as the collaborative model significantly reduces the number of large model inference calls while maintaining near-identical task accuracy, thereby substantially lowering computational resource consumption. Concurrently, the KAN-based small collaborative model markedly mitigates catastrophic forgetting, leading to significant accuracy improvements for long-tail data. The results reveal that KCM demonstrates superior performance across all metrics compared to MLP-based small collaborative models (MCM).

Figures (2)

• Figure 1: The overview of MLP and KAN.MLP performs polynomial calculation at the node, and the activation function is the same; KAN is calculated on the edge, the activation functions may be different, and the nodes of KAN only play the role of addition.
• Figure 2: The training process of KCM collaborate PLM. We input all the data into the judgment small model, input the samples with high confidence (representing suitable for small model prediction) into the small model, and input the samples with low confidence (representing suitable for large model prediction) into the large model, and combine the prompt modification of the small model with the knowledge distillation of the large model, and finally combine the prediction results of the two parts to get the result of the collaborative model.