LifeIR at the NTCIR-18 Lifelog-6 Task

TL;DR

This paper addresses semantic lifelog image retrieval under the LSAT task of NTCIR-18 Lifelog-6, where users issue textual queries describing scenes, actions, or events to locate corresponding images across long-term lifelogs. It proposes a multi-stage pipeline that filters blurred images, rewrites queries to clarify intent, extends the candidate set via temporally linked events, and reranks results with a multimodal large language model (MLLM) that offers stronger relevance judgment. The evaluation demonstrates the effectiveness of each stage and the overall pipeline, indicating improved retrieval accuracy for lifelog scenarios. The work advances practical lifelog search by enabling more accurate and context-aware image retrieval, benefiting users seeking specific moments across their lifelog histories.

Abstract

In recent years, sharing lifelogs recorded through wearable devices such as sports watches and GoPros, has gained significant popularity. Lifelogs involve various types of information, including images, videos, and GPS data, revealing users' lifestyles, dietary patterns, and physical activities. The Lifelog Semantic Access Task(LSAT) in the NTCIR-18 Lifelog-6 Challenge focuses on retrieving relevant images from a large scale of users' lifelogs based on textual queries describing an action or event. It serves users' need to find images about a scenario in the historical moments of their lifelogs. We propose a multi-stage pipeline for this task of searching images with texts, addressing various challenges in lifelog retrieval. Our pipeline includes: filtering blurred images, rewriting queries to make intents clearer, extending the candidate set based on events to include images with temporal connections, and reranking results using a multimodal large language model(MLLM) with stronger relevance judgment capabilities. The evaluation results of our submissions have shown the effectiveness of each stage and the entire pipeline.