Recall: Empowering Multimodal Embedding for Edge Devices

TL;DR

Recall is introduced, a novel on-device multimodal embedding system optimized for resource-limited mobile environments that achieves high-throughput, accurate retrieval by generating coarse-grained embeddings and leveraging query-based filtering for refined retrieval.

Abstract

Human memory is inherently prone to forgetting. To address this, multimodal embedding models have been introduced, which transform diverse real-world data into a unified embedding space. These embeddings can be retrieved efficiently, aiding mobile users in recalling past information. However, as model complexity grows, so do its resource demands, leading to reduced throughput and heavy computational requirements that limit mobile device implementation. In this paper, we introduce RECALL, a novel on-device multimodal embedding system optimized for resource-limited mobile environments. RECALL achieves high-throughput, accurate retrieval by generating coarse-grained embeddings and leveraging query-based filtering for refined retrieval. Experimental results demonstrate that RECALL delivers high-quality embeddings with superior throughput, all while operating unobtrusively with minimal memory and energy consumption.

Figures (23)

• Figure 1: MEM workflow and its application.
• Figure 2: Recall provides system-level service that remembers daily mobile interaction for recalling.
• Figure 3: Viewed image trace of mobile users.
• Figure 4: Demo of cross-modal retrieval.
• Figure 5: Throughput