Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM

Ching-Yun Ko, Pin-Yu Chen

TL;DR

The core functions of vLLM Hook are presented, an opensource plug-in to enable the programming of internal states for vLLM models and 3 use cases are demonstrated including prompt injection detection, enhanced retrieval-augmented retrieval (RAG), and activation steering.

Abstract

Modern artificial intelligence (AI) models are deployed on inference engines to optimize runtime efficiency and resource allocation, particularly for transformer-based large language models (LLMs). The vLLM project is a major open-source library to support model serving and inference. However, the current implementation of vLLM limits programmability of the internal states of deployed models. This prevents the use of popular test-time model alignment and enhancement methods. For example, it prevents the detection of adversarial prompts based on attention patterns or the adjustment of model responses based on activation steering. To bridge this critical gap, we present vLLM Hook, an opensource plug-in to enable the programming of internal states for vLLM models. Based on a configuration file specifying which internal states to capture, vLLM Hook provides seamless integration to vLLM and supports two essential features: passive programming and active programming. For passive programming, vLLM Hook probes the selected internal states for subsequent analysis, while keeping the model generation intact. For active programming, vLLM Hook enables efficient intervention of model generation by altering the selected internal states. In addition to presenting the core functions of vLLM Hook, in version 0, we demonstrate 3 use cases including prompt injection detection, enhanced retrieval-augmented retrieval (RAG), and activation steering. Finally, we welcome the community's contribution to improve vLLM Hook via https://github.com/ibm/vllm-hook.

vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM

TL;DR

The core functions of vLLM Hook are presented, an opensource plug-in to enable the programming of internal states for vLLM models and 3 use cases are demonstrated including prompt injection detection, enhanced retrieval-augmented retrieval (RAG), and activation steering.

Abstract

Modern artificial intelligence (AI) models are deployed on inference engines to optimize runtime efficiency and resource allocation, particularly for transformer-based large language models (LLMs). The vLLM project is a major open-source library to support model serving and inference. However, the current implementation of vLLM limits programmability of the internal states of deployed models. This prevents the use of popular test-time model alignment and enhancement methods. For example, it prevents the detection of adversarial prompts based on attention patterns or the adjustment of model responses based on activation steering. To bridge this critical gap, we present vLLM Hook, an opensource plug-in to enable the programming of internal states for vLLM models. Based on a configuration file specifying which internal states to capture, vLLM Hook provides seamless integration to vLLM and supports two essential features: passive programming and active programming. For passive programming, vLLM Hook probes the selected internal states for subsequent analysis, while keeping the model generation intact. For active programming, vLLM Hook enables efficient intervention of model generation by altering the selected internal states. In addition to presenting the core functions of vLLM Hook, in version 0, we demonstrate 3 use cases including prompt injection detection, enhanced retrieval-augmented retrieval (RAG), and activation steering. Finally, we welcome the community's contribution to improve vLLM Hook via https://github.com/ibm/vllm-hook.
Paper Structure (12 sections, 2 figures)

This paper contains 12 sections, 2 figures.

Table of Contents

  1. Overview of vLLM Hook
  2. What is vLLM Hook?
  3. Why do we need vLLM Hook?
  4. How do we use vLLM Hook?
  5. Core Functions of vLLM Hook
  6. Worker
  7. Analyzer
  8. Demonstrations of vLLM Hook
  9. How to Contribute to vLLM Hook?
  10. To contribute,
  11. Guidelines
  12. Acknowledgment

Figures (2)

  • Figure 1: Overview of vLLM Hook (https://github.com/ibm/vllm-hook), an opensource plug-in to enable flexible programming of internal states for models deployed on vLLM. With a customizable configuration file (Config), vLLM Hook supports two essential features: (1) Active Programming: vLLM Hook provides efficient intervention of model generation by altering the selected internal states; (2) Passive Programming: vLLM Hook probes and saves the selected internal states for subsequent analysis, while keeping the model generation intact.
  • Figure 3: An illustration of vLLM-Hook orchestration. vLLM-Hook is a lightweight wrapper around a native vLLM system and uses llm=HookLLM to initialize an LLM instance, llm.generate for response generation (model forward pass), and llm.analyze for optional in-model analysis