TikArt: Aperture-Guided Observation for Fine-Grained Visual Reasoning via Reinforcement Learning

TL;DR

TikArt presents aperture-guided observation to address fine-grained visual reasoning in multimodal LLMs by formulating reasoning as a Think-Aperture-Observe loop over regions of interest. It introduces two aperture actions (Zoom and Segmentation) and enforces a mandatory Observation after every action, turning local visual cues into persistent language memory. The method is trained with AGRPO, a two-stage GRPO-style reinforcement learning pipeline with task and aperture rewards, enabling the model to learn when and where to look without supervision on chain-of-thought. Empirical results across high-resolution benchmarks (V*, HR-Bench) and real-world multimodal tasks show consistent gains over the backbone and competitive performance against larger models, with interpretable aperture trajectories. Limitations include inference cost, reliance on external segmentation backbones (SAM2), and a compact action space, pointing to future work on more integrated front-ends, richer aperture actions, and scalable RL.

Abstract

We address fine-grained visual reasoning in multimodal large language models (MLLMs), where key evidence may reside in tiny objects, cluttered regions, or subtle markings that are lost under a single global image encoding. We introduce TikArt (Thinking Aperture), an aperture-guided agent that casts multi-step vision-language reasoning as a decision process over regions of interest. TikArt follows a Think-Aperture-Observe loop, alternating between language generation and two aperture actions: Zoom extracts rectangular crops, while Segment invokes SAM2 to obtain mask-based crops for irregular targets. After every action, the model must produce an explicit observation, turning local visual cues into persistent linguistic memory. Built on Qwen3-VL-8B, TikArt optimizes its reasoning policy with AGRPO, a GRPO-style reinforcement learning algorithm with a two-stage curriculum: it warms up segmentation actions and then jointly optimizes visual math, fine-grained VQA, and segmentation, using rewards that couple task success with purposeful aperture use. Experiments on V*, HR-Bench-4K/8K, MME-RealWorld-Lite, MMStar, RefCOCO, and ReasonSeg show consistent gains over the backbone and yield interpretable aperture trajectories for high-resolution reasoning.

Figures (9)

• Figure 1: The TikArt Framework. The model employs a Think-Aperture-Observe (TAO) Loop to iteratively perceive fine-grained visual details ($v_t$) before generating the final answer.
• Figure 2: TikArt performs aperture-guided observation across modalities. A single reasoning trajectory alternates between selecting regions of interest (RoIs) and describing what is seen there. In the example, the same sequence of aperture decisions supports a spatial VQA query (blue arrows) and a segmentation instruction (yellow arrows), while the dashed links indicate the underlying high-resolution image from which RoIs are extracted and refined. Segmentation masks are obtained by SAM2SAM-2.
• Figure 3: Overview of TikArt Framework. TikArt unifies token-level reasoning, visual aperture generation, and reinforcement learning signals. The model alternates between language generation and action invocation (Zoom/Segment) to form an aperture trajectory, while AGRPO provides task and action rewards to shape aperture-guided reasoning.
• Figure 4: Two-stage training pipeline of TikArt. Stage 1 warms up the Segmentation action head to produce reliable foreground masks. Stage 2 applies multi-task AGRPO over visual math, fine-grained VQA, and segmentation tasks to learn aperture-guided reasoning.
• Figure 5: Training dynamics of TikArt and GRPO-based variants under AGRPO. We plot (a) policy entropy, (b) the average number of aperture actions per trajectory, and (c) group reward over training steps. TikArt converges to a stable policy with moderate entropy and a reasonable aperture usage rate, while removing Observation leads to exploding entropy (above 2.0), uncontrolled aperture actions, and degraded reward. The w/o Segment Action variant tends to under-use apertures, whereas the w/o Zoom Action variant becomes more deterministic with lower entropy. The purple curve corresponds to a variant with different reward weights ($\beta_1 = 1.0, \beta_2 = 0.8$), illustrating the impact of reward design. For other benchmarks, we evaluate the w/o Observation variant using its checkpoint at step 260, before the onset of training divergence.