LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models

TL;DR

LENS addresses the challenge of translating long, real-world multimodal sensing into clinically meaningful narratives for mental health by first generating a large sensor–text QA dataset from EMA responses in a 90-day study with 258 participants. It then introduces a patch-based time-series encoder that projects raw sensor signals into the representation space of a large language model, enabling native multimodal reasoning and narrative generation. The approach is trained in two stages (encoder alignment and supervised fine-tuning) and evaluated with standard NLP metrics, a structured LLM-as-a-judge for symptom grounding, and a clinician user study, demonstrating superior narrative quality and clinical fidelity compared with time-series text and plot-based baselines. LENS achieves performance comparable to larger vision-based models while offering improved efficiency and direct, interpretable sensor-grounded narratives, suggesting a scalable path toward AI-assisted clinical decision support grounded in raw behavioral signals.

Abstract

Multimodal health sensing offers rich behavioral signals for assessing mental health, yet translating these numerical time-series measurements into natural language remains challenging. Current LLMs cannot natively ingest long-duration sensor streams, and paired sensor-text datasets are scarce. To address these challenges, we introduce LENS, a framework that aligns multimodal sensing data with language models to generate clinically grounded mental-health narratives. LENS first constructs a large-scale dataset by transforming Ecological Momentary Assessment (EMA) responses related to depression and anxiety symptoms into natural-language descriptions, yielding over 100,000 sensor-text QA pairs from 258 participants. To enable native time-series integration, we train a patch-level encoder that projects raw sensor signals directly into an LLM's representation space. Our results show that LENS outperforms strong baselines on standard NLP metrics and task-specific measures of symptom-severity accuracy. A user study with 13 mental-health professionals further indicates that LENS-produced narratives are comprehensive and clinically meaningful. Ultimately, our approach advances LLMs as interfaces for health sensing, providing a scalable path toward models that can reason over raw behavioral signals and support downstream clinical decision-making.

Figures (9)

• Figure 1: Illustration of the LENS idea. Mobile and wearable sensing signals, combined with a question, are passed to LENS, which produces a natural-language description. Clinicians can then view an interpretable snapshot of the user’s mental state instead of raw sensor streams.
• Figure 2: Overview of the narrative synthesis pipeline. EMA questions and responses are mapped to templates populated with corresponding frequency phrases. Subsequently, an LLM refines the text at both the item and summary levels (concatenated narratives) to enhance fluency and lexical diversity.
• Figure 3: LENS dataset construction pipeline. EMA responses are first converted into item-level and summary template narratives, which are then rewritten into fluent, enhanced narratives using Qwen2.5-14B. A multi-agent LLM-as-a-judge system conducts automatic quality control, routing failed cases back for regeneration until they satisfy all criteria. The final accepted narratives are then paired with paraphrased question variants to construct the QA datasets.
• Figure 4: LENS Architecture. The model accepts multimodal inputs consisting of description text (e.g., "Heart rate..."), instruction text (e.g., "Summarize the user's current mental well-being?") and raw time-series sensor streams (e.g., heart rate). The text is processed by a frozen LLM text embedder ($f_{\phi}^{emb}$), while time-series data is encoded by a trainable patch-based encoder ($f_{\theta}$). The resulting embeddings are concatenated into a unified sequence ($H$) and processed by the LLM backbone to generate a natural language response ($Y$).
• Figure 5: User Study: Comprehensiveness. Expert ratings compare how many symptoms each model’s narrative successfully covers.