Learnware of Language Models: Specialized Small Language Models Can Do Big

TL;DR

This paper extends the learnware paradigm to language models by building a dock of ~100 specialized $8\mathrm{B}$-parameter SLM learnwares across finance, healthcare, and mathematics, each paired with a compact specification. Users resolve tasks by privately generating task specifications and leveraging cosine-distance matching to select relevant learnwares, avoiding data exposure. Empirical results show per-task inference with learnwares can outperform base 8B models and rival several 70B+ LLMs, notably in finance and medical domains, while Beimingwu demonstrates practical deployment. The work highlights the value of decentralized, privacy-preserving, domain-specific expertise for resource-constrained AI applications and points to future directions in automating specification generation and broader platform integration.

Abstract

The learnware paradigm offers a novel approach to machine learning by enabling users to reuse a set of well-trained models for tasks beyond the models' original purposes. It eliminates the need to build models from scratch, instead relying on specifications (representations of a model's capabilities) to identify and leverage the most suitable models for new tasks. While learnware has proven effective in many scenarios, its application to language models has remained largely unexplored. At the same time, large language models (LLMs) have demonstrated remarkable universal question-answering abilities, yet they face challenges in specialized scenarios due to data scarcity, privacy concerns, and high computational costs, thus more and more specialized small language models (SLMs) are being trained for specific domains. To address these limitations systematically, the learnware paradigm provides a promising solution by enabling maximum utilization of specialized SLMs, and allowing users to identify and reuse them in a collaborative and privacy-preserving manner. This paper presents a preliminary attempt to apply the learnware paradigm to language models. We simulated a learnware system comprising approximately 100 learnwares of specialized SLMs with 8B parameters, fine-tuned across finance, healthcare, and mathematics domains. Each learnware contains an SLM and a specification, which enables users to identify the most relevant models without exposing their own data. Experimental results demonstrate promising performance: by selecting one suitable learnware for each task-specific inference, the system outperforms the base SLMs on all benchmarks. Compared to LLMs, the system outperforms Qwen1.5-110B, Qwen2.5-72B, and Llama3.1-70B-Instruct by at least 14% in finance domain tasks, and surpasses Flan-PaLM-540B (ranked 7th on the Open Medical LLM Leaderboard) in medical domain tasks.

Figures (6)

• Figure 1: Workflows of the Learnware Paradigm. (a) Workflow for Developers Uploading Learnware to the Learnware Dock System: ① LDS provides developers with the function $f$ for generating specifications. ② Developers generate a specification based on a model and data locally. ③ The specification and model are combined into a learnware. ④ The learnware is then submitted to the learnware dock system. (b) Workflow for Users Acquiring Learnware from the Learnware Dock System: ① User requests the acquisition of Learnware from LDS. ② LDS supplies the user with function $f$ for specification generation. ③ User generates a specification based on her data to represent her task requirements. ④ The requirement specification is submitted to the LDS. ⑤ LDS identifies and returns relevant learnware to the user based on the cosine similarity between user specifications and learnware specifications.
• Figure 2: Performances on financial LLM evaluation benchmark. The performance metrics are also normalized relative to Oracle. Detailed performance values are shown in \ref{['tab:finance']}.
• Figure 3: A diverse collection of SLM learnwares, though each of them falls behind LLMs for general unseen tasks, can surpass the capability boundaries of LLMs in specialized scenarios. For example, for the user task $T_1$ which can not be solved by LLM, the learnware $h_1$, though not originally designed for $T_1$, can handle it.
• Figure A4: The identification results of our method between user tasks and fine-tuned models. The number on the left of each user task denotes the rank of the selected fine-tuned model's performance among all models in solving this task.
• Figure A5: Performances our medical LLM evaluation benchmark. The performance metrics are also normalized relative to the "Oracle". Detailed performance values are shown in \ref{['tab:med']}.