Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

Xiaoding Lu, Zongyi Liu, Adian Liusie, Vyas Raina, Vineet Mudupalli, Yuwen Zhang, William Beauchamp

TL;DR

The paper tackles the high resource demands of trillion-parameter LLMs in chat applications by introducing Blended, a simple stochastic ensemble that selects one moderate-size model per turn to generate the next system response. By blending three 6-13B models, Blended can match or exceed the performance of a 175B+ model like ChatGPT while keeping inference costs comparable to a single small model. Large-scale CHAI platform A/B tests show Blended achieving higher user engagement and retention than the individual models and GPT-3.5, illustrating that model collaboration can deliver superior conversational quality with far lower resource usage. This work highlights a practical path toward more efficient, scalable, and engaging chat AIs through selective, per-turn model collaboration rather than simply increasing parameter counts.

Abstract

In conversational AI research, there's a noticeable trend towards developing models with a larger number of parameters, exemplified by models like ChatGPT. While these expansive models tend to generate increasingly better chat responses, they demand significant computational resources and memory. This study explores a pertinent question: Can a combination of smaller models collaboratively achieve comparable or enhanced performance relative to a singular large model? We introduce an approach termed "blending", a straightforward yet effective method of integrating multiple chat AIs. Our empirical evidence suggests that when specific smaller models are synergistically blended, they can potentially outperform or match the capabilities of much larger counterparts. For instance, integrating just three models of moderate size (6B/13B paramaeters) can rival or even surpass the performance metrics of a substantially larger model like ChatGPT (175B+ paramaters). This hypothesis is rigorously tested using A/B testing methodologies with a large user base on the Chai research platform over a span of thirty days. The findings underscore the potential of the "blending" strategy as a viable approach for enhancing chat AI efficacy without a corresponding surge in computational demands.

Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

TL;DR

The paper tackles the high resource demands of trillion-parameter LLMs in chat applications by introducing Blended, a simple stochastic ensemble that selects one moderate-size model per turn to generate the next system response. By blending three 6-13B models, Blended can match or exceed the performance of a 175B+ model like ChatGPT while keeping inference costs comparable to a single small model. Large-scale CHAI platform A/B tests show Blended achieving higher user engagement and retention than the individual models and GPT-3.5, illustrating that model collaboration can deliver superior conversational quality with far lower resource usage. This work highlights a practical path toward more efficient, scalable, and engaging chat AIs through selective, per-turn model collaboration rather than simply increasing parameter counts.

Abstract

In conversational AI research, there's a noticeable trend towards developing models with a larger number of parameters, exemplified by models like ChatGPT. While these expansive models tend to generate increasingly better chat responses, they demand significant computational resources and memory. This study explores a pertinent question: Can a combination of smaller models collaboratively achieve comparable or enhanced performance relative to a singular large model? We introduce an approach termed "blending", a straightforward yet effective method of integrating multiple chat AIs. Our empirical evidence suggests that when specific smaller models are synergistically blended, they can potentially outperform or match the capabilities of much larger counterparts. For instance, integrating just three models of moderate size (6B/13B paramaeters) can rival or even surpass the performance metrics of a substantially larger model like ChatGPT (175B+ paramaters). This hypothesis is rigorously tested using A/B testing methodologies with a large user base on the Chai research platform over a span of thirty days. The findings underscore the potential of the "blending" strategy as a viable approach for enhancing chat AI efficacy without a corresponding surge in computational demands.
Paper Structure (16 sections, 14 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 16 sections, 14 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chat AI approaches
  4. Generative system combination
  5. Blended
  6. Chat AI
  7. Ensembling
  8. Blended
  9. Evaluating Chat AIs
  10. User Retention
  11. User Engagement
  12. Experiments
  13. Experimental Set Up
  14. Experimental Results
  15. Future Work
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Model performance comparisons, setting the baseline as Pygmalion 6B. Each model is assigned to 5,000 unique new users, graphs report the day 30 retention and engagement improvement with respect to the baseline.
  • Figure 2: User Engagement
  • Figure 3: User Retention