What Sketch Explainability Really Means for Downstream Tasks

TL;DR

A lightweight and portable explainability solution - a seamless plugin that integrates effortlessly with any pre-trained model, elim-inating the need for re-training is proposed, and the centrepiece to the solution is a stroke-level attribution map that takes different forms when linked with downstream tasks.

Abstract

In this paper, we explore the unique modality of sketch for explainability, emphasising the profound impact of human strokes compared to conventional pixel-oriented studies. Beyond explanations of network behavior, we discern the genuine implications of explainability across diverse downstream sketch-related tasks. We propose a lightweight and portable explainability solution -- a seamless plugin that integrates effortlessly with any pre-trained model, eliminating the need for re-training. Demonstrating its adaptability, we present four applications: highly studied retrieval and generation, and completely novel assisted drawing and sketch adversarial attacks. The centrepiece to our solution is a stroke-level attribution map that takes different forms when linked with downstream tasks. By addressing the inherent non-differentiability of rasterisation, we enable explanations at both coarse stroke level (SLA) and partial stroke level (P-SLA), each with its advantages for specific downstream tasks.

Figures (12)

• Figure 1: We attribute explanations for individual strokes (stroke-level attribution) and their vector coordinate points (point-level attribution). Stroke-level attribution rasterises individual strokes (non-differentiably) to produce $n$-stroke images. Next, we sum the stroke images to get the complete sketch image used for downstream tasks. Point-level Attribution computes distance transform from stroke coordinates and thresholds to get the sketch image. Our explainability solution works without re-training for existing tasks like SBIR and sketch-to-photo generation and novel tasks like filtering noisy strokes for assisted drawing and adversarial attack by removing a small stroke.
• Figure 2: Coarse Stroke-level Attribution. Backpropagate gradients from raster sketch $\mathrm{X}$ to raster strokes $\mathcal{S}_{i}$, with weight $\omega_{i}$.
• Figure 3: Partial Stroke-level Attribution. Backpropagate gradients from raster sketch $\mathrm{X}$ to vector sequence of coordinates $\mathrm{V}$.
• Figure 4: Sketch attributions from stroke-level and point-level for image retrieval. High correlation of human-drawn stroke order with that from sketch-attributions (high$\to$low) indicate our sketch encoder gives more importance to salient strokes drawn early on.
• Figure 5: Assisted drawing via sketch healing (or filtering noisy strokes) using stroke attributions from SLA and P-SLA. This helps users having fear-to-sketch ("I can't sketch").