Read My Ears! Horse Ear Movement Detection for Equine Affective State Assessment

TL;DR

This work tackles the limited availability of annotated EquiFACS data for horses by comparing three pipelines for automated ear-related AU detection: a baseline optical-flow method (movDet), an Inflated 3D ConvNet + LSTM (I3D+LSTM), and VideoMAE + LSTM. On a dataset derived from rashidEquineFacialAction2020, VideoMAE+LSTM achieves the highest accuracy at 87.5% (with strong F1 performance), surpassing I3D+LSTM and movDet. The results demonstrate that transformer-based, self-supervised features can yield data-efficient AU detection in low-data settings, advancing automated equine welfare monitoring. The study also highlights practical deployment potential and calls for public release of code and data to foster broader applications across species.

Abstract

The Equine Facial Action Coding System (EquiFACS) enables the systematic annotation of facial movements through distinct Action Units (AUs). It serves as a crucial tool for assessing affective states in horses by identifying subtle facial expressions associated with discomfort. However, the field of horse affective state assessment is constrained by the scarcity of annotated data, as manually labelling facial AUs is both time-consuming and costly. To address this challenge, automated annotation systems are essential for leveraging existing datasets and improving affective states detection tools. In this work, we study different methods for specific ear AU detection and localization from horse videos. We leverage past works on deep learning-based video feature extraction combined with recurrent neural networks for the video classification task, as well as a classic optical flow based approach. We achieve 87.5% classification accuracy of ear movement presence on a public horse video dataset, demonstrating the potential of our approach. We discuss future directions to develop these systems, with the aim of bridging the gap between automated AU detection and practical applications in equine welfare and veterinary diagnostics. Our code will be made publicly available at https://github.com/jmalves5/read-my-ears.

Figures (30)

• Figure 1: Ear rotator action unit (EAD104) example.
• Figure 2: Horse facial muscles from wathanEquiFACSEquineFacial2015.
• Figure 3: Example EquiFACS AUs from askChangesEquineFacial2024.
• Figure 4: Dataset processing from videos in rashidEquineFacialAction2020.
• Figure 5: Pipeline for the baseline optical flow based ear movement detection (movDet).