Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TorchOpera: A Compound AI System for LLM Safety

Shanshan Han, Zijian Hu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Dimitris Stripelis, Zhaozhuo Xu, Chaoyang He

Abstract

We introduce TorchOpera, a compound AI system for enhancing the safety and quality of prompts and responses for Large Language Models. TorchOpera ensures that all user prompts are safe, contextually grounded, and effectively processed, while enhancing LLM responses to be relevant and high quality. TorchOpera utilizes the vector database for contextual grounding, rule-based wrappers for flexible modifications, and specialized mechanisms for detecting and adjusting unsafe or incorrect content. We also provide a view of the compound AI system to reduce the computational cost. Extensive experiments show that TorchOpera ensures the safety, reliability, and applicability of LLMs in real-world settings while maintaining the efficiency of LLM responses.

TorchOpera: A Compound AI System for LLM Safety

Abstract

We introduce TorchOpera, a compound AI system for enhancing the safety and quality of prompts and responses for Large Language Models. TorchOpera ensures that all user prompts are safe, contextually grounded, and effectively processed, while enhancing LLM responses to be relevant and high quality. TorchOpera utilizes the vector database for contextual grounding, rule-based wrappers for flexible modifications, and specialized mechanisms for detecting and adjusting unsafe or incorrect content. We also provide a view of the compound AI system to reduce the computational cost. Extensive experiments show that TorchOpera ensures the safety, reliability, and applicability of LLMs in real-world settings while maintaining the efficiency of LLM responses.
Paper Structure (12 sections, 5 figures, 1 table)

This paper contains 12 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. TorchOpera System Design
  4. TorchOpera Components
  5. TorchOpera Safety Detector
  6. TorchOpera Grounding
  7. TorchOpera Repair
  8. Implementation of TorchOpera
  9. Introduction of Base Model
  10. Efficient Deployment of Safety Detectors
  11. TorchOpera Compound System as a Service
  12. Conclusion

Figures (5)

  • Figure 1: System Overview.
  • Figure 2: Serving multiple LoRAs with a single GPU.
  • Figure 3: UI of defining TorchOpera as a service in our platform.
  • Figure 4: TorchOpera coumpound system as a service. Red lines indicate deployment flow. Purple lines indicate service flow.
  • Figure 5: Key steps of deploying TorchOpera.