Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
Yang Liu, Bingjie Yan, Tianyuan Zou, Jianqing Zhang, Zixuan Gu, Jianbing Ding, Xidong Wang, Jingyi Li, Xiaozhou Ye, Ye Ouyang, Qiang Yang, Ya-Qin Zhang
TL;DR
The paper addresses adapting large language models to private, sensitive-domain data without centralizing data or exposing proprietary models. It formulates the collaboration as a constrained optimization: $\max_{\theta_{\mathcal{T}}} \mathcal{F}(\theta_{\mathcal{L}}, \mathcal{D}, \theta_{\mathcal{S}}, \mathcal{T})$ subject to $M_{p}(\mathcal{D}, \theta_{\mathcal{S}}, \mathcal{I}_{\mathcal{T}}) \le \epsilon_{p}$, $M_{e}(\mathcal{D}, \theta_{\mathcal{S}}, \mathcal{I}_{\mathcal{T}}) \le \epsilon_{e}$, and $M_{L}(\theta_{\mathcal{L}}, \mathcal{J}_{\mathcal{T}}) \le \epsilon_{L}$. It surveys knowledge-transfer techniques across LM→SM, SM→LM, and collaborative inference, including distillation, data generation, adapters, federated learning, split learning, and retrieval-based approaches. It argues for industry-driven, application-focused benchmarks on real private datasets and outlines future directions with demonstrations in urban management, business intelligence, and personalized intelligence to bridge theory and practice.
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities, but they require vast amounts of data and computational resources. In contrast, smaller models (SMs), while less powerful, can be more efficient and tailored to specific domains. In this position paper, we argue that taking a collaborative approach, where large and small models work synergistically, can accelerate the adaptation of LLMs to private domains and unlock new potential in AI. We explore various strategies for model collaboration and identify potential challenges and opportunities. Building upon this, we advocate for industry-driven research that prioritizes multi-objective benchmarks on real-world private datasets and applications.