Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search

Kayhan Behdin, Qingquan Song, Sriram Vasudevan, Jian Sheng, Xiaojing Ma, Z Zhou, Chuanrui Zhu, Guoyao Li, Chanh Nguyen, Sayan Ghosh, Hejian Sang, Ata Fatahi Baarzi, Sundara Raman Ramachandran, Xiaoqing Wang, Qing Lan, Vinay Y S, Qi Guo, Caleb Johnson, Zhipeng Wang, Fedor Borisyuk

TL;DR

The paper tackles the challenge of deploying efficient semantic ranking with large language models in a high-throughput setting by focusing on a decoder-only small language model (SLM) and combining model pruning, context summarization, and systems-level serving optimizations. It demonstrates that structured pruning and selective transformer-block removal can achieve substantial parameter reductions with minimal loss in ranking quality, while RL-guided context summarization can reduce input length by about 93% with only a small drop in NDCG@10. When combined with a suite of offline and online serving optimizations—batch tokenization, decode-skip paths, memory- and GC-tuning, upfront prefill caching, dynamic scoring depth, and traffic shaping—the approach yields large throughput gains, including 4.6x offline and up to 2000 items/second per GPU online, enabling real-world deployment at LinkedIn. These results establish a practical pathway for scalable, low-latency semantic ranking with small LMs in industrial settings, delivering meaningful improvements in relevance metrics and user engagement.

Abstract

Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements. In this work, we present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn. Particularly, we discuss model compression techniques such as pruning that allow us to reduce the model size by up to $40\%$ while maintaining the accuracy. Additionally, we present context compression techniques that allow us to reduce the input context length by up to $10$x with minimal loss of accuracy. Finally, we present practical lessons from optimizing the serving infrastructure for deploying such a system on GPUs at scale, serving millions of requests per second. Taken together, this allows us to increase our system's throughput by $10$x in a real-world deployment, while meeting our quality bar.

Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search

TL;DR

The paper tackles the challenge of deploying efficient semantic ranking with large language models in a high-throughput setting by focusing on a decoder-only small language model (SLM) and combining model pruning, context summarization, and systems-level serving optimizations. It demonstrates that structured pruning and selective transformer-block removal can achieve substantial parameter reductions with minimal loss in ranking quality, while RL-guided context summarization can reduce input length by about 93% with only a small drop in NDCG@10. When combined with a suite of offline and online serving optimizations—batch tokenization, decode-skip paths, memory- and GC-tuning, upfront prefill caching, dynamic scoring depth, and traffic shaping—the approach yields large throughput gains, including 4.6x offline and up to 2000 items/second per GPU online, enabling real-world deployment at LinkedIn. These results establish a practical pathway for scalable, low-latency semantic ranking with small LMs in industrial settings, delivering meaningful improvements in relevance metrics and user engagement.

Abstract

Large Language Models (LLMs) have demonstrated impressive quality when applied to predictive tasks such as relevance ranking and semantic search. However, deployment of such LLMs remains prohibitively expensive for industry applications with strict latency and throughput requirements. In this work, we present lessons and efficiency insights from developing a purely text-based decoder-only Small Language Model (SLM) for a semantic search application at LinkedIn. Particularly, we discuss model compression techniques such as pruning that allow us to reduce the model size by up to while maintaining the accuracy. Additionally, we present context compression techniques that allow us to reduce the input context length by up to x with minimal loss of accuracy. Finally, we present practical lessons from optimizing the serving infrastructure for deploying such a system on GPUs at scale, serving millions of requests per second. Taken together, this allows us to increase our system's throughput by x in a real-world deployment, while meeting our quality bar.
Paper Structure (25 sections, 7 equations, 2 figures, 9 tables)

This paper contains 25 sections, 7 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Job Search
  4. Discussion of compression techniques
  5. Model compression via structured pruning
  6. Context compression via summarization
  7. Experimental Results
  8. Pruning Experiments
  9. Context Compression Experiments
  10. Prompt Engineering
  11. Reinforcement Learning
  12. Integrating Model and Context Compression
  13. Inference Benchmarking and Optimization
  14. Offline Benchmarking
  15. Serving Engine Optimization
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Overall pipeline of LinkedIn's Semantic Job Search.
  • Figure 2: Overall pipeline of context summarization with RL. We start with full item description, which is processed by the actor model to generate a summarized version. The summarized description is then combined with other tokens such as metadata before feeding into the SLM. The reward to the RL is calculated using the output of the SLM, coupled with the length of full and summarized description to update the parameters of the actor model. The parameters of the SLM are kept frozen throughout the process.