Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Embedded AI Companion System on Edge Devices

Rahul Gupta, Stephen D. H. Hsu

TL;DR

This work presents an embedded AI companion that runs entirely on edge devices by adopting an active–inactive memory paradigm: real-time dialogue leverages lightweight retrieval over short- and long-term memories to minimize latency, while extended periods of user inactivity trigger heavier extraction, consolidation, and updating of memories and user profiles to support long-term personalization. It also introduces a fully automated benchmark to holistically assess conversational quality, memory extraction, and personalization using synthetic users and LLM judges. Experiments with a quantized 7B model (int4) on edge hardware show memory-enabled edge performance surpassing a memoryless baseline and approaching GPT-3.5 with a 16k context, though a gap remains with GPT-5 in the cloud. The results demonstrate the feasibility of fully embedded AI companions and highlight the potential for hybrid edge–cloud approaches to close the remaining gap in performance. The framework and benchmark provide a path toward practical, private, low-latency AI companions on consumer edge devices.

Abstract

Computational resource constraints on edge devices make it difficult to develop a fully embedded AI companion system with a satisfactory user experience. AI companion and memory systems detailed in existing literature cannot be directly used in such an environment due to lack of compute resources and latency concerns. In this paper, we propose a memory paradigm that alternates between active and inactive phases: during phases of user activity, the system performs low-latency, real-time dialog using lightweight retrieval over existing memories and context; whereas during phases of user inactivity, it conducts more computationally intensive extraction, consolidation, and maintenance of memories across full conversation sessions. This design minimizes latency while maintaining long-term personalization under the tight constraints of embedded hardware. We also introduce an AI Companion benchmark designed to holistically evaluate the AI Companion across both its conversational quality and memory capabilities. In our experiments, we found that our system (using a very weak model: Qwen2.5-7B-Instruct quantized int4) outperforms the equivalent raw LLM without memory across most metrics, and performs comparably to GPT-3.5 with 16k context window.

Embedded AI Companion System on Edge Devices

TL;DR

This work presents an embedded AI companion that runs entirely on edge devices by adopting an active–inactive memory paradigm: real-time dialogue leverages lightweight retrieval over short- and long-term memories to minimize latency, while extended periods of user inactivity trigger heavier extraction, consolidation, and updating of memories and user profiles to support long-term personalization. It also introduces a fully automated benchmark to holistically assess conversational quality, memory extraction, and personalization using synthetic users and LLM judges. Experiments with a quantized 7B model (int4) on edge hardware show memory-enabled edge performance surpassing a memoryless baseline and approaching GPT-3.5 with a 16k context, though a gap remains with GPT-5 in the cloud. The results demonstrate the feasibility of fully embedded AI companions and highlight the potential for hybrid edge–cloud approaches to close the remaining gap in performance. The framework and benchmark provide a path toward practical, private, low-latency AI companions on consumer edge devices.

Abstract

Computational resource constraints on edge devices make it difficult to develop a fully embedded AI companion system with a satisfactory user experience. AI companion and memory systems detailed in existing literature cannot be directly used in such an environment due to lack of compute resources and latency concerns. In this paper, we propose a memory paradigm that alternates between active and inactive phases: during phases of user activity, the system performs low-latency, real-time dialog using lightweight retrieval over existing memories and context; whereas during phases of user inactivity, it conducts more computationally intensive extraction, consolidation, and maintenance of memories across full conversation sessions. This design minimizes latency while maintaining long-term personalization under the tight constraints of embedded hardware. We also introduce an AI Companion benchmark designed to holistically evaluate the AI Companion across both its conversational quality and memory capabilities. In our experiments, we found that our system (using a very weak model: Qwen2.5-7B-Instruct quantized int4) outperforms the equivalent raw LLM without memory across most metrics, and performs comparably to GPT-3.5 with 16k context window.
Paper Structure (33 sections, 4 figures, 4 tables)

This paper contains 33 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conversational, personalized, LLM-based, Long-term Memory Systems.
  4. Benchmarks and Synthetic Evaluation.
  5. Problem Formulation
  6. Method
  7. Overview
  8. Retrieval and Generation at Inference Time (Active)
  9. Extraction and Update (Inactive)
  10. Forgetting (Inactive)
  11. JSON validation and retries (Inactive)
  12. Greeting Phase (Active)
  13. Experimental Setup and Results
  14. Overview
  15. Comparison Baselines
  16. ...and 18 more sections

Figures (4)

  • Figure 3: Overview of System Architecture.
  • Figure 4: Coverage vs Input tokens.
  • Figure 5: Completeness vs Input tokens.
  • Figure 6: Information Present vs Input tokens