Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Lifelog Retrieval through Captioning-Enhanced Interpretation

Yu-Fei Shih, An-Zi Yen, Hen-Hsen Huang, Hsin-Hsi Chen

TL;DR

This work tackles the challenge of retrieving specific moments from first-person visual lifelogs using natural language queries. It introduces the Captioning-Integrated Visual Lifelog (CIVIL) retrieval system, which converts lifelog images into captions via large vision-language models and maps captions and queries into a shared embedding space for retrieval, avoiding task-specific training. The authors propose three captioning strategies—single, collective, and merged—and demonstrate that caption-based representations can outperform direct image embeddings, achieving up to P@10 = 0.73 with appropriate model pairings and text embeddings. They provide a textual lifelog caption dataset, perform extensive error analysis, and show that reranking with advanced LLMs like GPT-4o can further improve accuracy, highlighting practical implications for lifelog search and memory aids.

Abstract

People often struggle to remember specific details of past experiences, which can lead to the need to revisit these memories. Consequently, lifelog retrieval has emerged as a crucial application. Various studies have explored methods to facilitate rapid access to personal lifelogs for memory recall assistance. In this paper, we propose a Captioning-Integrated Visual Lifelog (CIVIL) Retrieval System for extracting specific images from a user's visual lifelog based on textual queries. Unlike traditional embedding-based methods, our system first generates captions for visual lifelogs and then utilizes a text embedding model to project both the captions and user queries into a shared vector space. Visual lifelogs, captured through wearable cameras, provide a first-person viewpoint, necessitating the interpretation of the activities of the individual behind the camera rather than merely describing the scene. To address this, we introduce three distinct approaches: the single caption method, the collective caption method, and the merged caption method, each designed to interpret the life experiences of lifeloggers. Experimental results show that our method effectively describes first-person visual images, enhancing the outcomes of lifelog retrieval. Furthermore, we construct a textual dataset that converts visual lifelogs into captions, thereby reconstructing personal life experiences.

Visual Lifelog Retrieval through Captioning-Enhanced Interpretation

TL;DR

This work tackles the challenge of retrieving specific moments from first-person visual lifelogs using natural language queries. It introduces the Captioning-Integrated Visual Lifelog (CIVIL) retrieval system, which converts lifelog images into captions via large vision-language models and maps captions and queries into a shared embedding space for retrieval, avoiding task-specific training. The authors propose three captioning strategies—single, collective, and merged—and demonstrate that caption-based representations can outperform direct image embeddings, achieving up to P@10 = 0.73 with appropriate model pairings and text embeddings. They provide a textual lifelog caption dataset, perform extensive error analysis, and show that reranking with advanced LLMs like GPT-4o can further improve accuracy, highlighting practical implications for lifelog search and memory aids.

Abstract

People often struggle to remember specific details of past experiences, which can lead to the need to revisit these memories. Consequently, lifelog retrieval has emerged as a crucial application. Various studies have explored methods to facilitate rapid access to personal lifelogs for memory recall assistance. In this paper, we propose a Captioning-Integrated Visual Lifelog (CIVIL) Retrieval System for extracting specific images from a user's visual lifelog based on textual queries. Unlike traditional embedding-based methods, our system first generates captions for visual lifelogs and then utilizes a text embedding model to project both the captions and user queries into a shared vector space. Visual lifelogs, captured through wearable cameras, provide a first-person viewpoint, necessitating the interpretation of the activities of the individual behind the camera rather than merely describing the scene. To address this, we introduce three distinct approaches: the single caption method, the collective caption method, and the merged caption method, each designed to interpret the life experiences of lifeloggers. Experimental results show that our method effectively describes first-person visual images, enhancing the outcomes of lifelog retrieval. Furthermore, we construct a textual dataset that converts visual lifelogs into captions, thereby reconstructing personal life experiences.

Paper Structure

This paper contains 30 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Captioning Retrieval System
  4. Overview
  5. Single caption method
  6. Collective caption method
  7. Merged caption method
  8. Frame Retrieval From User Query
  9. Experiments Setup
  10. Dataset and Evaluation Metrics
  11. Baseline Models
  12. Caption Models
  13. Single Caption Model
  14. Collective Caption Model
  15. Merged Caption Model
  16. ...and 15 more sections

Figures (5)

  • Figure 1: A sequence of first-person viewpoint images.
  • Figure 2: CIVIL Retrieval System Architecture. The system processes frames from multiple videos using three methods: (1) single-frame captions by an LVLM, (2) collective captions by a large video language model, and (3) fine- and coarse-grained captions by a Vision Embedding Model and LVLM. All captions and the user query are embedded in a shared text embedding space, and relevant frames are retrieved based on query-caption similarity.
  • Figure 3: Examples of Contextual Images Problem
  • Figure 4: Examples of errors across different methods.
  • Figure 5: Examples of Combination effect.