Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoJournaling: A Context-Aware Journaling System Leveraging MLLMs on Smartphone Screenshots

Tianyi Zhang, Shiquan Zhang, Le Fang, Hong Jia, Vassilis Kostakos, Simon D'Alfonso

TL;DR

This study proposes AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones, offering a novel approach to digital phenotyping.

Abstract

Journaling offers significant benefits, including fostering self-reflection, enhancing writing skills, and aiding in mood monitoring. However, many people abandon the practice because traditional journaling is time-consuming, and detailed life events may be overlooked if not recorded promptly. Given that smartphones are the most widely used devices for entertainment, work, and socialization, they present an ideal platform for innovative approaches to journaling. Despite their ubiquity, the potential of using digital phenotyping, a method of unobtrusively collecting data from digital devices to gain insights into psychological and behavioral patterns, for automated journal generation has been largely underexplored. In this study, we propose AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones. This system captures life events and corresponding emotions, offering a novel approach to digital phenotyping. We evaluated AutoJournaling by collecting screenshots every 3 seconds from three students over five days, demonstrating its feasibility and accuracy. AutoJournaling is the first framework to utilize seamlessly collected screenshots for journal generation, providing new insights into psychological states through digital phenotyping.

AutoJournaling: A Context-Aware Journaling System Leveraging MLLMs on Smartphone Screenshots

TL;DR

This study proposes AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones, offering a novel approach to digital phenotyping.

Abstract

Journaling offers significant benefits, including fostering self-reflection, enhancing writing skills, and aiding in mood monitoring. However, many people abandon the practice because traditional journaling is time-consuming, and detailed life events may be overlooked if not recorded promptly. Given that smartphones are the most widely used devices for entertainment, work, and socialization, they present an ideal platform for innovative approaches to journaling. Despite their ubiquity, the potential of using digital phenotyping, a method of unobtrusively collecting data from digital devices to gain insights into psychological and behavioral patterns, for automated journal generation has been largely underexplored. In this study, we propose AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones. This system captures life events and corresponding emotions, offering a novel approach to digital phenotyping. We evaluated AutoJournaling by collecting screenshots every 3 seconds from three students over five days, demonstrating its feasibility and accuracy. AutoJournaling is the first framework to utilize seamlessly collected screenshots for journal generation, providing new insights into psychological states through digital phenotyping.
Paper Structure (13 sections, 5 equations, 7 figures, 2 tables)

This paper contains 13 sections, 5 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method and Experiments
  3. AutoJournaling Framework
  4. Data Collection and Preprocessing
  5. Evaluation Framework
  6. Results & Discussion
  7. Performance of the AutoJournaling system
  8. Case Study
  9. Case I: Event analysis
  10. Case II: Feeling analysis
  11. Case III: Contextual information and logic analysis
  12. Privacy and Limitations
  13. Future Work and Conclusions

Figures (7)

  • Figure 2: Text analyzer by streams of image sets
  • Figure 3: Prompt for the image-to-text streamline of the AutoJournaling
  • Figure 4: Prompt for the video streamline of the AutoJournaling
  • Figure 5: An example comparison between the actual diary and the diary produced by AutoJournaling.
  • Figure : (a) The process of using LLMs to convert images into text descriptions in small sets, followed by journal generation
  • ...and 2 more figures