Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
Ziyao Wang, Muneeza Azmat, Ang Li, Raya Horesh, Mikhail Yurochkin
TL;DR
CoSD enables efficient test-time fusing of complementary LLM knowledge without retraining by using a draft model to generate initial tokens and an assistant model to verify in parallel. Token-level decisions are guided by either a Rule-Based or a Tree-Based verifier, both leveraging token probabilities to replace draft tokens when beneficial, with iterative regeneration until acceptance. Across six model pairs and multiple benchmarks, CoSD achieves up to 10% improvements in accuracy over state-of-the-art baselines while maintaining competitive efficiency and broad transferability, including scenarios with disparate capacities and differing tokenizers. The approach emphasizes interpretability and practicality for API-based use, illustrating a scalable path toward collaborative LLM deployment in diverse domains.
Abstract
Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by integrating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce a novel Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion at test time without requiring additional model training. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only enhances knowledge fusion but also improves inference efficiency, is transferable across domains and models, and offers greater explainability. Experimental results demonstrate that CoSD improves accuracy by up to 10\% across benchmarks compared to existing methods, providing a scalable and effective solution for LLM-based applications