Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure

TL;DR

This work introduces two distinct security-level training protocols: white-box and black-box, enhancing the SG-ZSL paradigm’s adaptability and robustness and efficiency across various setups, including stringent black-box training protocol.

Abstract

With increasing concerns over data privacy and model copyrights, especially in the context of collaborations between AI service providers and data owners, an innovative SG-ZSL paradigm is proposed in this work. SG-ZSL is designed to foster efficient collaboration without the need to exchange models or sensitive data. It consists of a teacher model, a student model and a generator that links both model entities. The teacher model serves as a sentinel on behalf of the data owner, replacing real data, to guide the student model at the AI service provider's end during training. Considering the disparity of knowledge space between the teacher and student, we introduce two variants of the teacher model: the omniscient and the quasi-omniscient teachers. Under these teachers' guidance, the student model seeks to match the teacher model's performance and explores domains that the teacher has not covered. To trade off between privacy and performance, we further introduce two distinct security-level training protocols: white-box and black-box, enhancing the paradigm's adaptability. Despite the inherent challenges of real data absence in the SG-ZSL paradigm, it consistently outperforms in ZSL and GZSL tasks, notably in the white-box protocol. Our comprehensive evaluation further attests to its robustness and efficiency across various setups, including stringent black-box training protocol.

Figures (6)

• Figure 1: In traditional ZSL approaches, real data is necessitated to establish the visual-semantic association. Conversely, SG-ZSL introduces a teacher model, which acts as a data sentinel, enabling the execution of ZSL tasks without the need for direct access to real data.
• Figure 2: Differences between the Omniscient and the Quasi-omniscient teacher.
• Figure 3: The overarching paradigm for both black-box and white-box protocols. In the white-box protocol, the generator accesses teacher weights during training, whereas in the black-box protocol, only output guidance from the teacher is utilized.
• Figure 4: Epoch analysis for unseen accuracy. 'Ver': label verification. 'R': regularization term.
• Figure 5: The t-SNE visualization on AWA1 and aPY. All experiments are simulated under white-box protocol, with the synthetic features in (a) and (b) generated from generators that follow the omniscient teacher (indicated with *), and those in (c) and (d) generated from generators that follow the quasi-omniscient teacher.