CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation

TL;DR

CoSense-LLM introduces an edge-first pipeline that converts continuous multimodal sensor streams into compact semantic tokens and coordinates with LLMs under explicit latency, energy, bandwidth, and privacy constraints. The method combines SenseFusion for multimodal encoding, Edge-RAG for local grounding, PromptRouter for cost- and uncertainty-aware cooperation, and Secure Execution for data minimization, supported by edge-optimized kernels, quantization, and federated personalization. Key contributions include a validated Edge-RAG grounding strategy, an uncertainty-aware routing policy, and extensive ablations showing reductions in latency, energy, and output contradictions while preserving explanation quality. The work demonstrates that semantics-at-the-edge with principled cloud–edge cooperation can achieve robust, private, and scalable LLM deployments in interference-prone environments, enabling practical on-site reasoning for sensing-driven applications.

Abstract

We present CoSense-LLM, an edge-first framework that turns continuous multimodal sensor streams (for example Wi-Fi CSI, IMU, audio, RFID, and lightweight vision) into compact, verifiable semantic tokens and coordinates with large language models under explicit latency, energy, bandwidth, and privacy constraints. CoSense-LLM has four parts: (i) SenseFusion, a lightweight encoder that aligns sensor embeddings with language and compresses them into short discrete code sequences; (ii) Edge-RAG, a local hybrid retrieval layer that grounds generation in site specific policies and notes; (iii) PromptRouter, a cost and uncertainty aware policy that selects edge only generation, edge plus retrieval, or compact cloud escalation; and (iv) Secure Execution, an auditable redaction path that enforces data minimization so raw waveforms never leave the device. The system works with modern serving optimizations, including paged or streaming KV caches, FlashAttention style kernels, speculative decoding, and quantized LoRA adapters, and supports on device personalization and federated updates under non IID drift. Across home, office, and clinic deployments, CoSense-LLM delivers grounded explanations while meeting tight service level objectives: it sustains sub second (p95) end to end latency on edge dominant paths, reduces inter tier token and bandwidth costs by preferring local retrieval grounded responses, and preserves privacy by transmitting only discrete codes and redacted metadata. Ablations show that Edge-RAG improves factual consistency and reduces contradictions, calibrated uncertainty enables selective abstention and controlled escalations, and KV plus decoding accelerators lower energy per decision. The results support an edge first design that treats semantics, privacy, and predictable latency as co equal goals for large model deployments in interference prone environments.

Figures (8)

• Figure 1: System overview
• Figure 2: Task performance (macro-F1) across datasets (HL/LO/HM) and methods.
• Figure 3: CDF of end-to-end latency under the Moderate link for Edge-Only, Edge+RAG, and Escalate.
• Figure 4: Energy per decision across datasets and methods. Lower is better.
• Figure 5: ROC for unknown-event detection (TS protocol): CoSense-LLM vs. edge classifier baseline.