Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model

TL;DR

Toolink introduces a two-stage framework that enables small open-source language models to use external tools by first constructing a task-specific toolkit and then solving queries via chain-of-solving (CoS). The CoS process splits reasoning into CoS-Planning and CoS-Calling, enabling transparent tool selection and code-based tool invocation, trained via the CoS-GPT dataset to produce LLaMA-CoS. Across BIG-bench tasks, Toolink with LLaMA-CoS matches ChatGPT's CoS ability and surpasses chain-of-thought baselines, while generalizing to unseen tasks and using generic toolkits. The framework offers a privacy-preserving offline solution with potential for scalable deployment on open-source models, and it provides an explicit dataset and methodology to extend tool-using capabilities beyond proprietary LLMs.

Abstract

Large Language Models (LLMs) have demonstrated remarkable progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptability, necessitating a valid method that leverages smaller, open-sourced models. In this paper, we introduce Toolink, a comprehensive framework that performs task-solving by first creating a toolkit and then integrating the planning and calling of tools through a chain-of-solving (CoS) approach. We first validate the efficacy of Toolink in harnessing the model's creativity and CoS ability on ChatGPT. Subsequently, we curate CoS-GPT, a chain-of-solving dataset designed for tool-using, and finetune the LLaMA-7B model. It results in LLaMA-CoS, a powerful open-source model with advanced tool-planning and tool-calling capabilities. Evaluation of diverse tasks from BIG-bench demonstrates its CoS ability matches that of ChatGPT while its performance surpasses the chain-of-thought approach. Further studies highlight the generalization of LLaMA-CoS to unseen tasks and showcase its capability in using toolkits not explicitly tailored for the target task, affirming its robustness in real-world scenarios.

Figures (17)

• Figure 1: An illustration of Toolink, which decomposes tasks via toolkit creation and resolves queries through Chain-of-Solving (CoS). Toolink can be adapted to open-source LLaMA for enhanced tool usage.
• Figure 2: A problem solving chain of Toolink pipeline. We show an example from task Navigate. Toolink first creates a toolkit generally applicable to the task, and then approaches the specific query through CoS, which involves planning and calling of the created tools.
• Figure 3: The improvement of performance when code generation data points are involved during training.
• Figure 4: Detailed error analysis of Alpaca and LLaMA-7B regarding CoS-Planning and CoS-Calling.
• Figure 5: Case Studies on the diverse CoS-Calling patterns in the main experiment.