Approximately optimal domain adaptation with Fisher's Linear Discriminant

TL;DR

This work introduces a Fisher's Linear Discriminant (FLD)–based, data-adaptive domain adaptation framework that convexly combines a source-task average classifier with a target-task classifier to improve performance under limited target data. The authors derive an analytically tractable risk expression under a generative model and provide a computable approximation for selecting the optimal convex weight $\alpha^*$ based on asymptotic distributions of projection vectors. Through simulations and three physiological prediction tasks (EEG/ECG), they demonstrate that the approximately optimal classifier often outperforms both the average-source and target classifiers, particularly in low-data regimes, and they discuss practical considerations such as privacy, computational cost, and visualization of projection vectors. Limitations include the two-class setting and a single global $\alpha$, with future directions pointing toward multi-class extensions and multimodal source modeling to capture more complex task distributions.

Abstract

We propose a class of models based on Fisher's Linear Discriminant (FLD) in the context of domain adaptation. The class is the convex combination of two hypotheses: i) an average hypothesis representing previously seen source tasks and ii) a hypothesis trained on a new target task. For a particular generative setting we derive the optimal convex combination of the two models under 0-1 loss, propose a computable approximation, and study the effect of various parameter settings on the relative risks between the optimal hypothesis, hypothesis i), and hypothesis ii). We demonstrate the effectiveness of the proposed optimal classifier in the context of EEG- and ECG-based classification settings and argue that the optimal classifier can be computed without access to direct information from any of the individual source tasks. We conclude by discussing further applications, limitations, and possible future directions.

Figures (8)

• Figure 1: A geometric illustration of the generative assumptions, information constraints, and the model class under study. All vectors are unit vectors. The red dots represent the data from the target distribution, the red arrow represents an estimate of the projection vector for the target distribution, the grey arrows represent source projection vectors, the blue arrow represents the average-source projection vector, and the green line interpolating between the blue and red arrows represents the possible end points of a convex combination of the red and blue arrows.
• Figure 2: Validating our proposed approximation by comparing the approximated analytical accuracies and empirical accuracies and optimal convex coefficients $\alpha^\ast$ for different amounts of target training data $n$ and number of source tasks $J$.
• Figure 3: Studying the effect of using plug-in estimates (left) and the effect of varying different generative model parameters (center, right) on the expected accuracy of the average-source, target, and optimal classifiers and on the optimal convex coefficient.
• Figure 4: Balanced accuracy and relevant convex coefficients (top) and relative performance of the optimal and target classifiers (bottom) for the MATB-II cognitive load classification task.
• Figure 5: Balanced accuracy and relevant convex coefficients (top) and relative performance of the optimal and target classifiers on a per-participant basis (bottom) for the Mental Math EEG-based stress classification task.