SMART-Vision: Survey of Modern Action Recognition Techniques in Vision

TL;DR

This survey addresses the problem of vision-based human action recognition by introducing the SMART-Vision taxonomy that reveals how modern HAR methods blend multiple architectures and modalities to form hybrid approaches. It surveys a wide spectrum of deep-learning techniques—from single-frame CNNs and two-stream models to 3D CNNs, GCNs, motion-based methods, and Transformer-based models—and highlights their extensions, hybridizations, and cross-modal transfers. A central contribution is the open-HAR analysis, detailing open-set and open-world evaluation paradigms and proposing robust protocol discussions to standardize future work. The paper also provides an in-depth discussion of datasets, challenges, and future directions, emphasizing efficiency, generalization, and the potential of vision-language and multimodal fusion to advance HAR in real-world scenarios.

Abstract

Human Action Recognition (HAR) is a challenging domain in computer vision, involving recognizing complex patterns by analyzing the spatiotemporal dynamics of individuals' movements in videos. These patterns arise in sequential data, such as video frames, which are often essential to accurately distinguish actions that would be ambiguous in a single image. HAR has garnered considerable interest due to its broad applicability, ranging from robotics and surveillance systems to sports motion analysis, healthcare, and the burgeoning field of autonomous vehicles. While several taxonomies have been proposed to categorize HAR approaches in surveys, they often overlook hybrid methodologies and fail to demonstrate how different models incorporate various architectures and modalities. In this comprehensive survey, we present the novel SMART-Vision taxonomy, which illustrates how innovations in deep learning for HAR complement one another, leading to hybrid approaches beyond traditional categories. Our survey provides a clear roadmap from foundational HAR works to current state-of-the-art systems, highlighting emerging research directions and addressing unresolved challenges in discussion sections for architectures within the HAR domain. We provide details of the research datasets that various approaches used to measure and compare goodness HAR approaches. We also explore the rapidly emerging field of Open-HAR systems, which challenges HAR systems by presenting samples from unknown, novel classes during test time.

Figures (12)

• Figure 1: SMART-Vision Venn Diagram. SMART-Vision diagram illustrates the formation of hybrid approaches that transcend the traditional categories in Section \ref{['sec:Deep learning approaches']}, including Two-Stream Networks (T-SNs) (Subsection \ref{['subsec:two-stream']}, Table \ref{['Tab:TwoStream-based-models']}), 3D Convolutional Networks (Subsection \ref{['subsec:3D CNNs']}, Table \ref{['Tab:3DCN-models']}), Graph Convolutional Networks (3DCN)(Subsection \ref{['subsec:GCNs']}, Table \ref{['Tab:GCNs-based-models']}), Motion Networks (MNs) (Subsection \ref{['subsec:Motion models']}, Table \ref{['Tab:MNs-models']}), Transformer Networks (TN)(Subsection \ref{['sec:Transformer Models']}, Table \ref{['Tab:Transformer-table']}), and Hybrid Networks (Subsection \ref{['sec:Hybrid Models']}). The SMART-Vision Taxonomy does not show some subsections, including additional novel work Subsection \ref{['sec:Additional Recent Novel Work']}. Note: The relative sizes of the shapes in the diagram do not indicate the volume of research: the purpose is to depict the categorization and interrelation of these sub-areas.
• Figure 2: A chronological overview of recent representative work in HAR. The chronological overview extends the work of Zhu et al. zhu2020comprehensive. The papers listed here represent major advancements in HAR; we discuss many more in Section 3 that could not be shown here due to display limitations.
• Figure 3: The two-stream architecture for video classification. The spatial stream (framed by a solid red border) illustrates a single image passing through a Convolutional Neural Network to a softmax layer. The temporal stream (delineated by a dashed blue border) shows stacked frames transforming into an optical flow map and passing through a Convolutional Neural Network and softmax layer. Both streams culminate in a joint diagram representing the fusion of their softmax scores into final class probabilities.
• Figure 4: Difference between GCN and CNN.
• Figure 5: The diagram shows different strategies for structuring convolution operations. It includes: (a) An input skeleton frame with body joints (blue dots) and filter receptive fields (red dashed circles). (b) A uniform labeling strategy with all nodes in a neighborhood sharing the same label (green). (c) A distance partitioning strategy separates the root node (green) and its neighbors (blue). (d) Spatial configuration partitioning where nodes are classified based on their proximity to the skeletal gravity center (black cross), with closer nodes (blue) and farther nodes (yellow) relative to the root node (green) yan2018spatial.