EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions

TL;DR

This work introduces EdgeWisePersona, a synthetic yet realism-grounded dataset and benchmark for reconstructing structured smart-home user routines from multi-session natural-language interactions, with a focus on edge-device deployment. Profiles combine routinely activated actions with contextual triggers, and sessions are generated by a large language model and validated by humans, enabling rigorous evaluation of on-device, privacy-preserving personalization. The study compares compact edge models to large foundation models, finding a substantial performance gap: large models achieve strong routine reconstruction across triggers, actions, and device parameters, while edge models struggle, especially for action prediction and numeric/categorical fields. The dataset, open-source tooling, and benchmark protocol provide a realistic testbed for developing and evaluating privacy-respecting, on-device AI that can learn and adapt directly on user-owned devices, advancing practical smart-home personalization.

Abstract

This paper introduces a novel dataset and evaluation benchmark designed to assess and improve small language models deployable on edge devices, with a focus on user profiling from multi-session natural language interactions in smart home environments. At the core of the dataset are structured user profiles, each defined by a set of routines - context-triggered, repeatable patterns of behavior that govern how users interact with their home systems. Using these profiles as input, a large language model (LLM) generates corresponding interaction sessions that simulate realistic, diverse, and context-aware dialogues between users and their devices. The primary task supported by this dataset is profile reconstruction: inferring user routines and preferences solely from interactions history. To assess how well current models can perform this task under realistic conditions, we benchmarked several state-of-the-art compact language models and compared their performance against large foundation models. Our results show that while small models demonstrate some capability in reconstructing profiles, they still fall significantly short of large models in accurately capturing user behavior. This performance gap poses a major challenge - particularly because on-device processing offers critical advantages, such as preserving user privacy, minimizing latency, and enabling personalized experiences without reliance on the cloud. By providing a realistic, structured testbed for developing and evaluating behavioral modeling under these constraints, our dataset represents a key step toward enabling intelligent, privacy-respecting AI systems that learn and adapt directly on user-owned devices.

Figures (3)

• Figure 1: Architecture of the dataset generation and benchmarking pipeline. The process is divided into two main parts: dataset generation (left) and benchmarking (right). Yellow boxes represent immutable input definitions: user routines and personality traits. Green boxes indicate dynamic values. Blue boxes denote models processing responsible for generating new sessions or inferring routines. Orange boxes correspond to outputs: generated sessions and inferred user routines. In the benchmark phase, models are evaluated on their ability to reconstruct structured user routines from the generated session data.
• Figure 2: Comparison of model performance on the routine prediction task using exact-match accuracy (left) and Jaccard similarity (right). Results are grouped by model size. Exact-match reflects full structural correctness, while Jaccard similarity captures partial overlap between predicted and reference routines.
• Figure 3: Model accuracy on trigger (left) and action (right) prediction. Scores are computed using exact-match evaluation against the best-matching ground-truth routine.