Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains

Yuqi Xiong, Chunyi Peng, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Yukun Yan, Shuo Wang, Yu Gu, Ge Yu

TL;DR

Lang2Act is proposed, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains and designs a two-stage Reinforcement Learning (RL)-based training framework to support this mechanism.

Abstract

Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.

Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains

TL;DR

Lang2Act is proposed, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains and designs a two-stage Reinforcement Learning (RL)-based training framework to support this mechanism.

Abstract

Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.
Paper Structure (23 sections, 14 equations, 17 figures, 10 tables)

This paper contains 23 sections, 14 equations, 17 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Enhancing Visual Reasoning Trajectories for Linguistic Tool Curation
  5. Optimizing Vision-Language Models through Linguistic Tool-Based Prompting
  6. Experimental Methodology
  7. Evaluation Result
  8. Overall Performance
  9. Ablation Study
  10. Effectiveness of Linguistic Tools in Visual Document Perception
  11. Case Study
  12. Conclusion
  13. Appendix
  14. License
  15. Additional Details of Datasets
  16. ...and 8 more sections

Figures (17)

  • Figure 1: Comparison between VRAG-RL and the Lang2Act framework.
  • Figure 2: The overview architecture of Lang2Act.
  • Figure 3: Quantitative analysis of image perception quality in relation to QA accuracy. We compute V-Precision (Figure \ref{['fig:V-precision']}) and V-Recall (Figure \ref{['fig:V-recall']}) for analysis by leveraging the perception rate on the golden region together with QA accuracy. The perception rate is calculated according to whether the model internal attention hit the golden region.
  • Figure 4: Case Study on SlideVQA. The red box indicates the ground truth region of the given image.
  • Figure 5: Distribution of sample difficulty before and after filtering in the Tool-Based Optimization training.
  • ...and 12 more figures