Towards the Resistance of Neural Network Watermarking to Fine-tuning

TL;DR

It is proved that when the input feature of a convolutional layer only contains low-frequency components, specific frequency components of the convolutional filter will not be changed by gradient descent during the fine-tuning process, where a revised Fourier transform is proposed to extract frequency components from the convolutional filter.

Abstract

This paper proves a new watermarking method to embed the ownership information into a deep neural network (DNN), which is robust to fine-tuning. Specifically, we prove that when the input feature of a convolutional layer only contains low-frequency components, specific frequency components of the convolutional filter will not be changed by gradient descent during the fine-tuning process, where we propose a revised Fourier transform to extract frequency components from the convolutional filter. Additionally, we also prove that these frequency components are equivariant to weight scaling and weight permutations. In this way, we design a watermark module to encode the watermark information to specific frequency components in a convolutional filter. Preliminary experiments demonstrate the effectiveness of our method.

Figures (5)

• Figure 1: The framework of the proposed watermark. We prove that the specific frequency componentsfn:comp_filter $\mathcal{F}^{(uv)}_\mathbf{W}$, which are obtained by conducting a revised discrete Fourier transform $\mathcal{T}(\cdot)$ on the convolutional filter $\mathbf{W}$, keep stable in the training process. Thus, these specific frequency components $\mathcal{F}^{(uv)}_\mathbf{W}$ are used as the robust watermark to fine-tuning. For clarity, we move low frequencies to the center of the spectrum map, and move high frequencies to corners of the spectrum map. Unless otherwise stated, in this paper, we visualize the frequency spectrum map in this manner.
• Figure 2: Forward propagation in the frequency domain (a) and forward propagation in the spatial domain (b). The convolution operation with a convolutional filter on the input feature $\mathbf{X}$ is essentially equivalent to a vector multiplication on the frequency components of the input.
• Figure 3: The architecture of the watermark module. The watermark module is connected in parallel to the backbone of the neural network. We extract the specific frequency components from the convolutional filters in the watermark module as the network's watermark.
• Figure 4: Visualization of the watermark. (a) shows the specific frequencies in the set $S'$ used as the watermark. (b) shows the feature maps when we apply the inverse discrete Fourier transform (IDFT) to some unit frequency components used as the watermark. The frequency components are extracted from a single channel of a $3 \times 3$ convolutional filter in the watermark module and the input feature map has a width and height of $9 \times 9$, so the set $S' = \{ (u, v) | u = 3i\ \text{or} \ v = 3j;\ i, j \in \{1, 2\} \}$. For clarity, we move low frequencies to the center of the spectrum map, and move high frequencies to corners of the spectrum map.
• Figure 5: Heatmaps showing the average norm of the change of the frequency components $\mathbb{E}_{d}[ \| \Delta \mathcal{F}_{\mathbf{W}_{d}}^{(uv)} \|]$ before and after fine-tuning at different frequencies $(u,v)$ over all convolutional filters. For clarity, we move low frequencies to the center of the spectrum map, and move high frequencies to corners of the spectrum map

Theorems & Definitions (12)

• Theorem 3.1
• Theorem 3.2
• Corollary 3.3
• Proposition 3.4
• Theorem 3.5
• Theorem 3.6
• Lemma A.1
• proof
• proof
• proof