Elytra: A Flexible Framework for Securing Large Vision Systems

TL;DR

A framework called ELYTRA is proposed to take insights for parameter-efficient fine-tuning and use low-rank adaptation (LoRA) to train a lightweight security patch (or patches), enabling us to dynamically patch large pre-existing vision systems as new vulnerabilities are discovered.

Abstract

Adversarial attacks have emerged as a critical threat to autonomous driving systems. These attacks exploit the underlying neural network, allowing small, almost invisible, perturbations to alter the behavior of such systems in potentially malicious ways, e.g., causing a traffic sign classification network to misclassify a stop sign as a speed limit sign. Prior work in hardening such systems against adversarial attacks has looked at fine-tuning of the system or adding additional pre-processing steps to the input pipeline. Such solutions either have a hard time generalizing, require knowledge of adversarial attacks during training, or are computationally undesirable. Instead, we propose a framework called ELYTRA to take insights for parameter-efficient fine-tuning and use low-rank adaptation (LoRA) to train a lightweight security patch (or patches), enabling us to dynamically patch large pre-existing vision systems as new vulnerabilities are discovered. We demonstrate that the ELYTRA framework can patch pre-trained large vision models to improve classification accuracy by up to 24.09% in the presence of adversarial examples.

Figures (4)

• Figure 1: Overview of Elytra. (a) Diverse adversarial attacks cause misclassifications in large vision models. (b)Elytra trains a lightweight LoRA adapter (security patch) per attack type while keeping all base model weights frozen. (c) Adapters compose additively into a single multi-threat defense.
• Figure 2: An illustration of adversarial perturbations applied to an example set of signs ("Clean"). N.B., that the Euclidean projection onto the feasibility set $\mathcal{S}$ in PGD keeps the distortions minimal at similar step sizes $\varepsilon$ when compared to FGSM. In the first row, we have a "Stop Sign", followed by "No Overtaking", and lastly "Roundabout".
• Figure 3: An illustration of accuracy versus parameter count of Elytra and adversarial hardening methods compared to the baseline model. The left graph is the accuracy of the model on non-adversarial data after Elytra or fine-tuning. The right graph is the accuracy of the model on adversarial data after Elytra (blue circle) or fine-tuning (red circle). The y-axis scale is shared to illustrate the model's vulnerability to adversarial inputs. In each graph, the higher and to the left, the better.
• Figure 4: Performance of Elytras as the quantity of Elytras composed increases. n represents the number of models generated per number of Elytras composed