Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On-Device LLMs for SMEs: Challenges and Opportunities

Jeremy Stephen Gabriel Yee, Pai Chet Ng, Zhengkui Wang, Ian McLoughlin, Aik Beng Ng, Simon See

TL;DR

The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles.

Abstract

This paper presents a systematic review of the infrastructure requirements for deploying Large Language Models (LLMs) on-device within the context of small and medium-sized enterprises (SMEs), focusing on both hardware and software perspectives. From the hardware viewpoint, we discuss the utilization of processing units like GPUs and TPUs, efficient memory and storage solutions, and strategies for effective deployment, addressing the challenges of limited computational resources typical in SME settings. From the software perspective, we explore framework compatibility, operating system optimization, and the use of specialized libraries tailored for resource-constrained environments. The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles. Such a structured review provides practical insights, contributing significantly to the community by enhancing the technological resilience of SMEs in integrating LLMs.

On-Device LLMs for SMEs: Challenges and Opportunities

TL;DR

The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles.

Abstract

This paper presents a systematic review of the infrastructure requirements for deploying Large Language Models (LLMs) on-device within the context of small and medium-sized enterprises (SMEs), focusing on both hardware and software perspectives. From the hardware viewpoint, we discuss the utilization of processing units like GPUs and TPUs, efficient memory and storage solutions, and strategies for effective deployment, addressing the challenges of limited computational resources typical in SME settings. From the software perspective, we explore framework compatibility, operating system optimization, and the use of specialized libraries tailored for resource-constrained environments. The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles. Such a structured review provides practical insights, contributing significantly to the community by enhancing the technological resilience of SMEs in integrating LLMs.

Paper Structure

This paper contains 13 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Resource-Constrained Environments
  3. Software Framework
  4. Operating System Compatibility
  5. Compute Unified Device Architecture
  6. Specialized Library
  7. Hardware Innovations
  8. High-Performance GPU and TPU
  9. Memory and Storage
  10. Deployment Efficiency
  11. Conclusion
  12. Acknowledgements
  13. About the Authors

Figures (1)

  • Figure 1: Operationalization of LLMs in SMEs Under Resource-Constrained Environments: This diagram illustrates the essential components involved in deploying LLMs within SMEs, focusing on RAG and fine-tuning processes, integrated with local SME databases. Key components for hardware innovations, including processing units and storage solutions, are emphasized in an orange box. Components related to software operations are outlined in a blue box. Areas where hardware and software considerations intersect are depicted in overlapping sections of the boxes.