Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch

TL;DR

This work addresses the lack of efficient inference for fine-grained sketch-based image retrieval (FG-SBIR) by showing that off-the-shelf photo-efficient networks underperform sketches and introducing two plug-in components to tailor them to sketch data. It presents SketchyNetV1, a cross-modal knowledge distillation framework that transfers semantic structure from a large teacher to a lightweight student, dramatically reducing FLOPs and parameters while preserving embedding quality. Building on this, SketchyNetV2 adds an abstraction-aware canvas-size selector trained with reinforcement learning to render sketches at dynamically chosen resolutions, achieving a further ~$3.28\times$ FLOPs reduction and near-competition-grade accuracy. Collectively, the approach enables practically deployable sketch-based retrieval on resource-constrained devices, maintaining accuracy while delivering substantial compute savings across multiple FG-SBIR datasets and even enabling faster sketch recognition tasks.

Abstract

As sketch research has collectively matured over time, its adaptation for at-mass commercialisation emerges on the immediate horizon. Despite an already mature research endeavour for photos, there is no research on the efficient inference specifically designed for sketch data. In this paper, we first demonstrate existing state-of-the-art efficient light-weight models designed for photos do not work on sketches. We then propose two sketch-specific components which work in a plug-n-play manner on any photo efficient network to adapt them to work on sketch data. We specifically chose fine-grained sketch-based image retrieval (FG-SBIR) as a demonstrator as the most recognised sketch problem with immediate commercial value. Technically speaking, we first propose a cross-modal knowledge distillation network to transfer existing photo efficient networks to be compatible with sketch, which brings down number of FLOPs and model parameters by 97.96% percent and 84.89% respectively. We then exploit the abstract trait of sketch to introduce a RL-based canvas selector that dynamically adjusts to the abstraction level which further cuts down number of FLOPs by two thirds. The end result is an overall reduction of 99.37% of FLOPs (from 40.18G to 0.254G) when compared with a full network, while retaining the accuracy (33.03% vs 32.77%) -- finally making an efficient network for the sparse sketch data that exhibit even fewer FLOPs than the best photo counterpart.

Figures (5)

• Figure 1: Our SketchyNetV1 compresses existing heavy FG-SBIR networks to deliver smaller models. Further enhanced via a canvas-selector module our SketchyNetV2 model minimises sketch-resolution dynamically to reduce FLOPs.
• Figure 2: Pilot study for varying canvas-size (see text above).
• Figure 3: Two stage framework. (Left) SketchyNetV1: A smaller student network ($F_{st}$) is trained from a larger pre-trained teacher ($F_T$) via KD (§ \ref{['sec:student_model']}). (Right) We progress to SketchyNetV2 by training a canvas-size selector directed by objectives of increasing performance and reducing compute. It takes sketch as a vector ($s_v$) and aims to predict an optimal canvas-size at which the sketch is rasterised ($s_r$). $s_r$ when processed by $F_{st}$ minimises overall FLOPS, while retaining accuracy of corresponding full-resolution sketch-image.
• Figure 4: Exemplar sketches at their optimal canvas-size
• Figure 5: Impact of varying $\lambda_\text{F}$ on Acc@1 and Acc@5