FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning

TL;DR

FineRS tackles ultra-small object reasoning and segmentation in high-resolution scenes with a two-stage reinforcement-learning framework that couples Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR) via a locate-informed retrospective reward. It introduces FineRS-4k, a UAV-based 4K dataset with instruction-guided text–mask annotations to benchmark fine-grained reasoning and pixel-level segmentation. The method leverages a frozen SAM2 for mask generation and achieves strong performance on instruction-guided segmentation and VQA tasks under data-scarce conditions, demonstrating data-efficient, fine-grained perception at high resolutions. These advances hold practical significance for remote sensing and surveillance applications requiring precise localization and reasoning about extremely small targets in cluttered scenes.

Abstract

Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities across a wide range of vision-language tasks. However, due to the restricted input resolutions, MLLMs face significant challenges in precisely understanding and localizing visual details in high-resolution images -- particularly when dealing with extra-small objects embedded in cluttered contexts. To address this issue, we propose \textsc{FineRS}, a two-stage MLLM-based reinforcement learning framework for jointly reasoning and segmenting extremely small objects within high-resolution scenes. \textsc{FineRS} adopts a coarse-to-fine pipeline comprising Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). Specifically, GSE performs instruction-guided reasoning to generate a textural response and a coarse target region, while LPR refines this region to produce an accurate bounding box and segmentation mask. To couple the two stages, we introduce a locate-informed retrospective reward, where LPR's outputs are used to optimize GSE for more robust coarse region exploration. % Additionally, we present \textsc{FineRS}-4k, a new dataset for evaluating MLLMs on attribute-level reasoning and pixel-level segmentation on subtle, small-scale targets in complex high-resolution scenes. Experimental results on \textsc{FineRS}-4k and public datasets demonstrate that our method consistently outperforms state-of-the-art MLLM-based approaches on both instruction-guided segmentation and visual reasoning tasks.

Figures (7)

• Figure 1: Overview of our benchmark. (a) The innermost ring shows three instruction types. The middle ring presents the mask size distribution within each type. The outermost ring breaks each type down into four attribute categories: color, shape, position, and others. (b) Visualization of six representative examples, each illustrating a different attribute category (color, shape, position, others) across the three instruction types in the outermost ring.
• Figure 2: Framework overview of FineRS. (a) During training, we design specific reward functions to train GSE and LPR, where the LPR is optimized first and adopted to form a retrospective reward to enhance the coarse region accuracy of GSE. (b) During inference, GSE takes a high-resolution image and user instructions as input and produces an answer and a coarse region containing referred objects. Then, LPR processes the instruction and coarse region to generate the object box and adopts SAM2 ravi2024sam2 to generate the final mask. (c) To unify VQA and segmentation into a single MLLM, we design a multi-task reward pool and assign the items to supervise GSE and LPR.
• Figure 3: Visual results on Open-ended VQA (OVQA), Multiple-choice VQA (MVQA), and Instruction-guided Segmentation (IS). The listed images are sampled from FineRS-4k test set.
• Figure A1: (a) Distribution of mask sizes across three subsets, where the x-axis indicates the ratio of the mask area to the entire image. (b) Spatial division of the image into three regions --- center, middle, and border—based on the location of the mask. (c) Distribution of mask locations across three subsets.
• Figure A2: More visual results on Open-ended VQA (OVQA), Multiple-choice VQA (MVQA), and Instruction-guided Segmentation (IS). The listed images are sampled from FineRS-4k test set.