Orthogonal Activation with Implicit Group-Aware Bias Learning for Class Imbalance
Sukumar Kishanthan, Asela Hevapathige
TL;DR
This work tackles class imbalance in deep learning by introducing OGAB, an activation layer that injects inductive biases through an orthogonal feature transform and implicit group-aware bias learning. The orthogonal transform, realized via a learned skew-symmetric construct and Cayley transform to yield an orthogonal matrix $Q$, preserves minority-class information during embedding updates. Per-sample group gating assigns context-specific biases, producing $V_i$ that modulate embeddings to improve class separability, while the final output blends the transformed features with nonlinear activation and scaling. Empirical results on six datasets show OGAB surpasses traditional and several learnable activations in F1-score and balanced accuracy, with modest parameter overhead and comparable runtime to sampling-based imbalance techniques, suggesting practical applicability and potential for transfer-learning extensions.
Abstract
Class imbalance is a common challenge in machine learning and data mining, often leading to suboptimal performance in classifiers. While deep learning excels in feature extraction, its performance still deteriorates under imbalanced data. In this work, we propose a novel activation function, named OGAB, designed to alleviate class imbalance in deep learning classifiers. OGAB incorporates orthogonality and group-aware bias learning to enhance feature distinguishability in imbalanced scenarios without explicitly requiring label information. Our key insight is that activation functions can be used to introduce strong inductive biases that can address complex data challenges beyond traditional non-linearity. Our work demonstrates that orthogonal transformations can preserve information about minority classes by maintaining feature independence, thereby preventing the dominance of majority classes in the embedding space. Further, the proposed group-aware bias mechanism automatically identifies data clusters and adjusts embeddings to enhance class separability without the need for explicit supervision. Unlike existing approaches that address class imbalance through preprocessing data modifications or post-processing corrections, our proposed approach tackles class imbalance during the training phase at the embedding learning level, enabling direct integration with the learning process. We demonstrate the effectiveness of our solution on both real-world and synthetic imbalanced datasets, showing consistent performance improvements over both traditional and learnable activation functions.