Attacking All Tasks at Once Using Adversarial Examples in Multi-Task Learning

TL;DR

This work investigates adversarial robustness in multi-task learning (MTL) and introduces Dynamic Gradient Balancing Attack (DGBA), a framework that optimizes perturbations to simultaneously degrade all tasks by dynamically balancing task gradients. DGBA reformulates the attack as a multi-task optimization, relaxes it to a linear program, and yields an exact, coordinate-wise solution that can be combined with existing white-box attacks. Extensive experiments on NYUv2 and Tiny-Taskonomy show DGBA outperforming strong baselines across various sharing patterns and even when models are adversarially trained, while revealing a trade-off: increased parameter sharing improves task accuracy but raises transferability and vulnerability to attacks. Overall, the study highlights the need to balance accuracy and robustness in MTL design and suggests extensions to large multimodal systems as a future direction.

Abstract

Visual content understanding frequently relies on multi-task models to extract robust representations of a single visual input for multiple downstream tasks. However, in comparison to extensively studied single-task models, the adversarial robustness of multi-task models has received significantly less attention and many questions remain unclear: 1) How robust are multi-task models to single task adversarial attacks, 2) Can adversarial attacks be designed to simultaneously attack all tasks in a multi-task model, and 3) How does parameter sharing across tasks affect multi-task model robustness to adversarial attacks? This paper aims to answer these questions through careful analysis and rigorous experimentation. First, we analyze the inherent drawbacks of two commonly-used adaptations of single-task white-box attacks in attacking multi-task models. We then propose a novel attack framework, Dynamic Gradient Balancing Attack (DGBA). Our framework poses the problem of attacking all tasks in a multi-task model as an optimization problem that can be efficiently solved through integer linear programming. Extensive evaluation on two popular MTL benchmarks, NYUv2 and Tiny-Taxonomy, demonstrates the effectiveness of DGBA compared to baselines in attacking both clean and adversarially trained multi-task models. Our results also reveal a fundamental trade-off between improving task accuracy via parameter sharing across tasks and undermining model robustness due to increased attack transferability from parameter sharing.

Figures (7)

• Figure 1: Attack effectiveness in terms of Average Relative Performance (ARP) as defined in Eq.\ref{['equ:arp']} (y-axis, higher-the-better) for each task when applying Single, Total, and the proposed DGBA attacks on NYUv2 (a three-task dataset). ARP is a generic metric that reflects how much the task performance has degraded after an attack regardless of the diverse metrics used in different tasks. The attack variants are built on APGD. Segm: semantic segmentation; Norm: normal prediction; Dept: depth estimation.
• Figure 2: Attack performance comparisons in terms of ARP averaged over 25 multi-task models trained on NYUv2 with Deeplab-ResNet34. The adapted attacks and DGBA variants are built on (a) FGSM, (b) PGD, (c) APGD, and (d) Auto-SAGA. The perturbation bound $\epsilon$ ranges from 1 to 16.
• Figure 3: Attack performance comparisons in terms of ARP on NYUv2 with MobileNetV2 similar to Figure \ref{['fig:NYUv2-resnet-overall']}.
• Figure 4: Attack performance comparisons between WGD and DGBA on NYUv2 with Deeplab-ResNet34. The left figures compare the average ARP over 25 multi-task models while the right ones illustrate the attack effectiveness on each task.
• Figure 5: The relationship between the levels of parameter sharing in multi-task models (x-axis) and the attack transferability (z-axis). The y-axis represents APGD variants Single-x.