Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

Tommaso Galliena, Stefano Rosa, Tommaso Apicella, Pietro Morerio, Alessio Del Bue, Lorenzo Natale

Abstract

Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints, hindering the ability of embodied agents to construct consistent semantic representations over time. Previous methods resolved inconsistencies using offline multi-view aggregation or multi-stage pipelines that decouple exploration, data association, and caption learning, with limited capacity to reason over previously observed objects. In this paper, we introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy within a single autoregressive framework. The model processes the current RGB observation, a top-down explored map, and an object-level episodic memory serialized into object-level tokens, ensuring persistent object identity and semantic consistency across extended sequences. To train the model in a self-supervised manner, we collect a dataset in photorealistic 3D environments using a disagreement-based policy and a pseudo-captioning model that enforces consistency across multi-view caption histories. Extensive evaluation on a manually annotated object-level test set, demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models, while enabling scalable performance through a compact scene representation. Code, model weights, and data are available at https://hsp-iit.github.io/epos-vlm/.

Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning

Abstract

Vision-Language Models (VLMs) often yield inconsistent descriptions of the same object across viewpoints, hindering the ability of embodied agents to construct consistent semantic representations over time. Previous methods resolved inconsistencies using offline multi-view aggregation or multi-stage pipelines that decouple exploration, data association, and caption learning, with limited capacity to reason over previously observed objects. In this paper, we introduce a unified, memory-augmented Vision-Language agent that simultaneously handles data association, object captioning, and exploration policy within a single autoregressive framework. The model processes the current RGB observation, a top-down explored map, and an object-level episodic memory serialized into object-level tokens, ensuring persistent object identity and semantic consistency across extended sequences. To train the model in a self-supervised manner, we collect a dataset in photorealistic 3D environments using a disagreement-based policy and a pseudo-captioning model that enforces consistency across multi-view caption histories. Extensive evaluation on a manually annotated object-level test set, demonstrate improvements of up to +11.86% in standard captioning scores and +7.39% in caption self-similarity over baseline models, while enabling scalable performance through a compact scene representation. Code, model weights, and data are available at https://hsp-iit.github.io/epos-vlm/.
Paper Structure (26 sections, 4 equations, 7 figures, 8 tables)

This paper contains 26 sections, 4 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Embodied Persistent Object Semantics Visual Language Model (EPOS-VLM)
  4. Visual Observations and Object Identifiers
  5. Episodic Object Memory Representation and Tokenization
  6. Data Association
  7. Training EPOS-VLM
  8. Experimental Setup
  9. Results
  10. Pseudo-Captioning Evaluation
  11. Object-level Captioning and Semantic Consistency
  12. Data Association and Scalability
  13. Action Policy Evaluation
  14. Qualitative results
  15. Conclusion
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Memory-driven multi-view exploration progressively resolves ambiguous object captions into a consistent object-level description. (a) the agent predicts a caption for the observed object; (b) a different caption is predicted from a different viewpoint; (c) a consistent caption is predicted based on the episodic object memory; (d) the predicted caption for the object remains consistent.
  • Figure 2: Overview of the proposed EPOS-VLM model. Structured episodic memory, RGB observations (with detected object bounding boxes and IDs overlayed), and top-down maps are jointly encoded and processed by a Vision–Language transformer, which autoregressively outputs object association, object-level captions, and actions.
  • Figure 3: Mean and standard deviation of inference time (left) and memory occupancy (right) averaged over HM3D ramakrishnan2021habitatmatterport3ddatasethm3d test episodes for EPOS-VLM and the dense point cloud baseline Galliena_2025_ICCV. KEY: Point cloud, Ours
  • Figure 4: Comparison of object captions predicted by a VLM baseline (Qwen3-VL) and our method along successive viewpoints of an exploration. Mistakes highlighted in red.
  • Figure 5: Mean and standard deviation of memory token count (left) and number of objects (right) per step, averaged over all HM3D test episodes.
  • ...and 2 more figures