Advancing TDFN: Precise Fixation Point Generation Using Reconstruction Differences

TL;DR

The paper addresses pixel-level fixation point generation for Task-Driven Fixation Networks (TDFN) by replacing reinforcement learning with a reconstruction-difference mechanism. An internal image reconstruction module predicts a high-resolution template $x^{*}$, and fixation points are drawn from regions with the largest reconstruction difference via a SaliencyMap; the model is trained end-to-end using a combined loss that includes classification, reconstruction, and fixation terms with $(\alpha,\beta)=(0.2,0.1)$. Empirical results on MNIST show that fixation points generated from reconstruction differences (FPG2) yield higher classification accuracy and require fewer fixations compared to the RL-based approach (FPG1), with dynamic termination further reducing computation while maintaining accuracy. The approach enables finer, pixel-precise localization on high-resolution inputs and supports a dynamic, cost-effective patching strategy in Transformer-based architectures, marking a practical advance in attention-guided visual processing for tasks with limited high-resolution access.

Abstract

Wang and Wang (2025) proposed the Task-Driven Fixation Network (TDFN) based on the fixation mechanism, which leverages low-resolution information along with high-resolution details near fixation points to accomplish specific visual tasks. The model employs reinforcement learning to generate fixation points. However, training reinforcement learning models is challenging, particularly when aiming to generate pixel-level accurate fixation points on high-resolution images. This paper introduces an improved fixation point generation method by leveraging the difference between the reconstructed image and the input image to train the fixation point generator. This approach directs fixation points to areas with significant differences between the reconstructed and input images. Experimental results demonstrate that this method achieves highly accurate fixation points, significantly enhances the network's classification accuracy, and reduces the average number of required fixations to achieve a predefined accuracy level.

Figures (3)

• Figure 1: TDFN architecture.
• Figure 2: Different maps output by TDFN. The first column shows the original input image, the second column displays the reconstructed image, the third column represents the absolute difference between the reconstructed image and the input image, and the last column depicts the saliency map generated by the fixation point generator.
• Figure 3: Visualization of Fixation Points by FPG2. The first column shows the original input images. The second column presents the low-resolution images (odd rows) and the reconstructed images generated by TDFN using only the low-resolution inputs (even rows). The third through ninth columns sequentially display the fixation points generated by the Fixation Point Generator (FPG) (odd rows, represented as light squares) and the reconstructed images generated by TDFN using both low-resolution inputs and high-resolution ROIs centered at the fixation points (even rows). The reconstructed images demonstrate how the inclusion of fixation points introduces supplementary information, enhancing the reconstruction quality.