Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CiMRAG: Cim-Aware Domain-Adaptive and Noise-Resilient Retrieval-Augmented Generation for Edge-Based LLMs

Shih-Hsuan Chiu, Ming-Syan Chen

TL;DR

The paper tackles the challenge of personalized edge LLMs using RAG on Computing-in-Memory (CiM) hardware, where rapid growth of profile data and hardware-induced noise degrade retrieval accuracy. It introduces TONEL, a label-free framework consisting of NATO (noise-aware task-oriented optimization) and PGM (pseudo-label generation) to produce CiM-friendly, task-specific embeddings that remain robust under CiM perturbations and across multiple domains. By projecting high-dimensional encoder outputs to CiM-friendly vectors (e.g., $64$-dimensional, 8-bit) and training with a CiM-aware loss that accounts for hardware noise $\eta \sim \mathcal{N}(0, \sigma_v)$, TONEL achieves superior MIPS-based retrieval (Acc@1, Prec@5, nDCG@5) on LaMP Movie/Rating benchmarks, even in high-noise conditions, and yields strong downstream classification improvements with edge LLMs. The work demonstrates practical gains for edge-based personalization, reducing data movement and avoiding on-device fine-tuning while enabling domain-adaptive retrieval in resource-constrained environments.

Abstract

Personalized virtual assistants powered by large language models (LLMs) on edge devices are attracting growing attention, with Retrieval-Augmented Generation (RAG) emerging as a key method for personalization by retrieving relevant profile data and generating tailored responses. However, deploying RAG on edge devices faces efficiency hurdles due to the rapid growth of profile data, such as user-LLM interactions and recent updates. While Computing-in-Memory (CiM) architectures mitigate this bottleneck by eliminating data movement between memory and processing units via in-situ operations, they are susceptible to environmental noise that can degrade retrieval precision. This poses a critical issue in dynamic, multi-domain edge-based scenarios (e.g., travel, medicine, and law) where both accuracy and adaptability are paramount. To address these challenges, we propose Task-Oriented Noise-resilient Embedding Learning (TONEL), a framework that improves noise robustness and domain adaptability for RAG in noisy edge environments. TONEL employs a noise-aware projection model to learn task-specific embeddings compatible with CiM hardware constraints, enabling accurate retrieval under noisy conditions. Extensive experiments conducted on personalization benchmarks demonstrate the effectiveness and practicality of our methods relative to strong baselines, especially in task-specific noisy scenarios.

CiMRAG: Cim-Aware Domain-Adaptive and Noise-Resilient Retrieval-Augmented Generation for Edge-Based LLMs

TL;DR

The paper tackles the challenge of personalized edge LLMs using RAG on Computing-in-Memory (CiM) hardware, where rapid growth of profile data and hardware-induced noise degrade retrieval accuracy. It introduces TONEL, a label-free framework consisting of NATO (noise-aware task-oriented optimization) and PGM (pseudo-label generation) to produce CiM-friendly, task-specific embeddings that remain robust under CiM perturbations and across multiple domains. By projecting high-dimensional encoder outputs to CiM-friendly vectors (e.g., -dimensional, 8-bit) and training with a CiM-aware loss that accounts for hardware noise , TONEL achieves superior MIPS-based retrieval (Acc@1, Prec@5, nDCG@5) on LaMP Movie/Rating benchmarks, even in high-noise conditions, and yields strong downstream classification improvements with edge LLMs. The work demonstrates practical gains for edge-based personalization, reducing data movement and avoiding on-device fine-tuning while enabling domain-adaptive retrieval in resource-constrained environments.

Abstract

Personalized virtual assistants powered by large language models (LLMs) on edge devices are attracting growing attention, with Retrieval-Augmented Generation (RAG) emerging as a key method for personalization by retrieving relevant profile data and generating tailored responses. However, deploying RAG on edge devices faces efficiency hurdles due to the rapid growth of profile data, such as user-LLM interactions and recent updates. While Computing-in-Memory (CiM) architectures mitigate this bottleneck by eliminating data movement between memory and processing units via in-situ operations, they are susceptible to environmental noise that can degrade retrieval precision. This poses a critical issue in dynamic, multi-domain edge-based scenarios (e.g., travel, medicine, and law) where both accuracy and adaptability are paramount. To address these challenges, we propose Task-Oriented Noise-resilient Embedding Learning (TONEL), a framework that improves noise robustness and domain adaptability for RAG in noisy edge environments. TONEL employs a noise-aware projection model to learn task-specific embeddings compatible with CiM hardware constraints, enabling accurate retrieval under noisy conditions. Extensive experiments conducted on personalization benchmarks demonstrate the effectiveness and practicality of our methods relative to strong baselines, especially in task-specific noisy scenarios.
Paper Structure (11 sections, 12 equations, 3 figures, 4 tables)

This paper contains 11 sections, 12 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. RAG on CiM devices with MIPS-based retrieval
  4. Proposed Method
  5. Noise-aware Task-oriented Optimization Strategy
  6. Pseudo-Label Generation Mechanism
  7. Empirical Experiments
  8. Experimental Setup
  9. Experimental Results
  10. Conclusion
  11. Acknowledgment

Figures (3)

  • Figure 1: A schematic depiction of RAG for edge LLMs with CiM devices; the workflow is inspired by qin2024robust. CiM employs MIPS to retrieve top-ranked documents, which are combined with the user query to enable the LLM to generate personalized responses. This paper focuses on enhancing the text embedding model to produce task-aware, noise-resilient CiM-friendly embeddings for MIPS in domain-specific edge environments.
  • Figure 2: The overview of the proposed TONEL framework. It focuses on optimizing the projection model to produce task-aware, noise-resilient CiM-friendly vectors (in gold) suitable for MIPS in domain-specific edge environments.
  • Figure 3: A schematic depiction of PGM for generating pseudo task labels used in TONEL.