V2A-Mark: Versatile Deep Visual-Audio Watermarking for Manipulation Localization and Copyright Protection

TL;DR

V2A-Mark is proposed to address the limitations of current video tampering forensics, and can embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio, enabling precise manipulation localization and copyright protection.

Abstract

AI-generated video has revolutionized short video production, filmmaking, and personalized media, making video local editing an essential tool. However, this progress also blurs the line between reality and fiction, posing challenges in multimedia forensics. To solve this urgent issue, V2A-Mark is proposed to address the limitations of current video tampering forensics, such as poor generalizability, singular function, and single modality focus. Combining the fragility of video-into-video steganography with deep robust watermarking, our method can embed invisible visual-audio localization watermarks and copyright watermarks into the original video frames and audio, enabling precise manipulation localization and copyright protection. We also design a temporal alignment and fusion module and degradation prompt learning to enhance the localization accuracy and decoding robustness. Meanwhile, we introduce a sample-level audio localization method and a cross-modal copyright extraction mechanism to couple the information of audio and video frames. The effectiveness of V2A-Mark has been verified on a visual-audio tampering dataset, emphasizing its superiority in localization precision and copyright accuracy, crucial for the sustainable development of video editing in the AIGC video era.

Figures (7)

• Figure 1: Overall Framework of our proposed V$\mathbf{^{2}}$A-Mark. We embed pre-defined visual localization watermark $\mathbf{W}_{loc}$, copyright watermark $\mathbf{w}_{cop}$ and audio versatile watermark $\mathbf{w}^{\prime}_{cop}$ into the original video frames and audio to produce $\mathbf{V}_{con}$ and $\mathbf{A}_{con}$. If undergoing malicious tampering, we can still extract exact copyright $\hat{\mathbf{w}}_{cop}$, visual tampered masks $\hat{\mathbf{M}}_{vis}$ and audio tampered periods $\hat{\mathbf{m}}_{aud}$. Note that $\hat{\mathbf{w}}_{cop}$ is obtained via our cross-modal extraction mechanism, combining $\mathbf{w}^a_{cop}$ and $\mathbf{w}^v_{cop}$.
• Figure 2: Details of the network structure and training process of the proposed V$^2$A-Mark. We design the temporal alignment and fusion module (TAFM) and degradation prompt learning (DPL) to enhance the robustness and fidelity of our method.
• Figure 3: Details of the proposed temporal alignment and fusion module (TAFM). It aligns the supporting frames $\mathbf{I}^{(k-1)}_{ori}$, $\mathbf{I}^{(k+1)}_{ori}$ to the reference frame $\mathbf{I}^{(k)}_{ori}$.
• Figure 4: Details of the proposed degradation prompt learning mechanism. It fuses the intrinsic image features $\mathbf{F}_v$/$\mathbf{F}_b$ with the learnable prompt components $\mathbf{P}_v$/$\mathbf{P}_b$ adaptively.
• Figure 5: Localization accuracy comparison with our V$^2$A-Mark and other localization methods PSCC-Net liu2022pscc, EditGuard zhang2023editguard. Our method can predict more accurate and clearer tampered masks. We also present our container and tampered videos.