Feature Engineering is Not Dead: Reviving Classical Machine Learning with Entropy, HOG, and LBP Feature Fusion for Image Classification

TL;DR

The paper revisits classical machine learning for image classification by extending Permutation Entropy (PE) to 2D images and fusing it with HOG and LBP to form a 780-dimensional handcrafted feature vector for SVM classification. This interpretable pipeline achieves competitive results across Fashion-MNIST (≈91%), KMNIST (≈94%), EMNIST (≈93%), and CIFAR-10 (≈65%), highlighting the value of entropy-based descriptors alongside traditional texture/shape cues. It also demonstrates that combining handcrafted features with lightweight deep embeddings can further improve performance (e.g., CIFAR-10 reaching 93.10% with fusion) while reducing training cost compared to full end-to-end CNN fine-tuning. The results support the viability of hybrid deep+handcrafted architectures and reinforce that feature engineering remains a practical, efficient, and interpretable approach in modern vision tasks.

Abstract

Feature engineering continues to play a critical role in image classification, particularly when interpretability and computational efficiency are prioritized over deep learning models with millions of parameters. In this study, we revisit classical machine learning based image classification through a novel approach centered on Permutation Entropy (PE), a robust and computationally lightweight measure traditionally used in time series analysis but rarely applied to image data. We extend PE to two-dimensional images and propose a multiscale, multi-orientation entropy-based feature extraction approach that characterizes spatial order and complexity along rows, columns, diagonals, anti-diagonals, and local patches of the image. To enhance the discriminatory power of the entropy features, we integrate two classic image descriptors: the Histogram of Oriented Gradients (HOG) to capture shape and edge structure, and Local Binary Patterns (LBP) to encode micro-texture of an image. The resulting hand-crafted feature set, comprising of 780 dimensions, is used to train Support Vector Machine (SVM) classifiers optimized through grid search. The proposed approach is evaluated on multiple benchmark datasets, including Fashion-MNIST, KMNIST, EMNIST, and CIFAR-10, where it delivers competitive classification performance without relying on deep architectures. Our results demonstrate that the fusion of PE with HOG and LBP provides a compact, interpretable, and effective alternative to computationally expensive and limited interpretable deep learning models. This shows a potential of entropy-based descriptors in image classification and contributes a lightweight and generalizable solution to interpretable machine learning in image classification and computer vision.

Figures (6)

• Figure 1: Visualization of PE from a time-series data: (a) a sample time series data; and (b) corresponding normalized PE with embedding dimension d = 3 and time delay $\tau$ =1. Note that in high irregularity domain PE is near 1 and in high regularity or predictable domain PE is near 0.
• Figure 2: Visualization of HOG from an image: (a) reference image (source: scikit-image); (b) corresponding HOG that nicely captures shape, outline and edges of the image; (c) Schematic visualization of 9 HOG Orientation Bins (0°–180°, unsigned; arrows = gradient directions). Note that a gradient at 90° (arrow pointing straight up) comes from a horizontal edge, while a gradient at 0°/180° (arrow along the x-axis) comes from a vertical edge; (d) plot of global HOG orientation histogram (9 orientation bins, 0°–180°) in which each bar aggregates the total gradient magnitude for orientations falling into its 20° bin.
• Figure 3: Visualization of LBP from an image: (a) reference image (source: scikit-image); (b) corresponding LBP which is good at encoding fine-scaled local texture.
• Figure 4: (a) Schematic overview of the proposed workflow pipeline. For each input image we extract permutation-entropy (PE) based features—computed along rows, columns, the diagonal and anti-diagonals, and on overlapping local patches. We combine them with row-/column-correlation based features, histogram of oriented gradients (HOG) and local binary pattern (LBP) image descriptors. The four feature sets are concatenated into a single feature vector and classified with a support vector machine (SVM). (b) Detailed illustration of the PE-based feature-extraction steps on a (6 × 6) toy image. PE is first evaluated for every row and column, followed by the diagonal (top-left → bottom-right) and anti-diagonal (top-right → bottom-left) directions. Note that some off-centre diagonals and anti-diagonals (towards corner) PEs are not computed because their reduced length may yield unreliable PE features or statistics. Finally, we compute patch-wise PE with (2 × 2) patches and with a stride length of 2 pixels. See Section \ref{['sec:FE_extraction']} for a complete description of the feature-extraction pipeline.
• Figure 5: Visualization of extracted features from a single Fashion-MNIST sample: (a) original 28×28 grayscale image; (b) patch-wise permutation-entropy (PE) heatmap computed over 4×4 windows with stride 2; (c) local binary pattern (LBP) texture map using P=16, R=2; (d) Histogram of Oriented Gradients (HOG) visualization with 4×4 pixels per cell; and (e) finally a normalized, concatenated feature vector used as input to the SVM classifier.