AutoLife: Automatic Life Journaling with Smartphones and LLMs

TL;DR

AutoLife tackles automatic life journaling by fusing multimodal smartphone sensors with LLM/VLM reasoning to generate long-duration semantic journals without user input. A multi-layer pipeline decomposes data into motion and location contexts, converts them into concise textual representations, and then synthesizes comprehensive journals via LLMs, aided by a duty-cycle data collection strategy to manage energy use. The system is evaluated on a real-life Hong Kong dataset and demonstrates high journal quality with competitive metrics (e.g., BERTScore, chrF) and no hallucinations, while also offering clear cost savings and practical deployment guidance. The work provides a publicly available benchmark and shows that combining map-based and WiFi-based location cues with structured prompts enables robust, privacy-conscious life journaling on commodity smartphones, paving the way for downstream personalized analytics and applications.

Abstract

This paper introduces a novel mobile sensing application - life journaling - designed to generate semantic descriptions of users' daily lives. We present AutoLife, an automatic life journaling system based on commercial smartphones. AutoLife only inputs low-cost sensor data (without photos or audio) from smartphones and can automatically generate comprehensive life journals for users. To achieve this, we first derive time, motion, and location contexts from multimodal sensor data, and harness the zero-shot capabilities of Large Language Models (LLMs), enriched with commonsense knowledge about human lives, to interpret diverse contexts and generate life journals. To manage the task complexity and long sensing duration, a multilayer framework is proposed, which decomposes tasks and seamlessly integrates LLMs with other techniques for life journaling. This study establishes a real-life dataset as a benchmark and extensive experiment results demonstrate that AutoLife produces accurate and reliable life journals.

Figures (13)

• Figure 1: Life journaling application.
• Figure 2: Existing lifelogging solutions. Left shows a user wears SenseCam glogger_2013 while right shows two digital diary applications, i.e., Day One dayone_journal_app and Journal apple_2023_journal.
• Figure 3: AutoLife overview.
• Figure 4: Examples of detecting location contexts with address and places. Results are from Google Maps Geocoding and Places API, respectively. The left side shows the map segments centered at corresponding locations.
• Figure 5: Examples of detecting location contexts by analyzing map images with VLM. The results are generated from GPT-4o OpenAI2024GPT4o and input images are the maps in Figure \ref{['fig:location:other']}.