RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization

TL;DR

RelayFormer tackles visual manipulation localization across images and videos by addressing resolution diversity and the image–video modality gap. It introduces Input Unification and Global-Local Relay Attention (GLRA), which uses a small set of GLR tokens and 4D Rotary Positional Embeddings to propagate global cues while preserving local forensic traces, with a parameter-efficient backbone supported by LoRA-style adaptations. A lightweight Query-based Mask Decoder predicts precise manipulation masks, trained with a BCE plus edge-focused loss. Empirical results on diverse image and video benchmarks show state-of-the-art performance with strong efficiency, demonstrating practical applicability for real-time forensic analysis.

Abstract

Visual manipulation localization (VML) aims to identify tampered regions in images and videos, a task that has become increasingly challenging with the rise of advanced editing tools. Existing methods face two main issues: resolution diversity, where resizing or padding distorts forensic traces and reduces efficiency, and the modality gap, as images and videos often require separate models. To address these challenges, we propose RelayFormer, a unified framework that adapts to varying resolutions and modalities. RelayFormer partitions inputs into fixed-size sub-images and introduces Global-Local Relay (GLR) tokens, which propagate structured context through a global-local relay attention (GLRA) mechanism. This enables efficient exchange of global cues, such as semantic or temporal consistency, while preserving fine-grained manipulation artifacts. Unlike prior methods that rely on uniform resizing or sparse attention, RelayFormer naturally scales to arbitrary resolutions and video sequences without excessive overhead. Experiments across diverse benchmarks demonstrate that RelayFormer achieves state-of-the-art performance with notable efficiency, combining resolution adaptivity without interpolation or excessive padding, unified modeling for both images and videos, and a strong balance between accuracy and computational cost. Code is available at: https://github.com/WenOOI/RelayFormer.

Figures (9)

• Figure 1: Illustration of several common types of visual manipulation, including splicing, copy-move, and inpainting. (a) Examples of manipulated regions and their corresponding boundaries generated by these methods. (b) A schematic illustration highlighting the need for both local and global information to accurately localize manipulated regions.
• Figure 2: Overview of our proposed framework, which consists of three main components. First, the input image or video is partitioned into unified local sub-images without interpolation, preserving fine-grained spatial details. Second, we propose the GLRA module to achieve efficient global information propagation. Finally, a carefully designed lightweight mask decoder efficiently produces the prediction masks. For clarity, the positional encoding components are omitted from the figure.
• Figure 3: Detailed architecture of the proposed Global-Local Relay Attention (GLRA) module.
• Figure 4: Visual qualitative results for image and video scenarios.
• Figure 5: Robustness analysis results of the model under common perturbations.