Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tracking Mouse from Incomplete Body-Part Observations and Deep-Learned Deformable-Mouse Model Motion-Track Constraint for Behavior Analysis

Olaf Hellwich, Niek Andresen, Katharina Hohlbaum, Marcus N. Boon, Monika Kwiatkowski, Simon Matern, Patrik Reiske, Henning Sprekeler, Christa ThöneReineke, Lars Lewejohann, Huma Ghani Zada, Michael Brück, Soledad Traverso

TL;DR

This work tackles the challenge of occlusion-driven gaps in $3D$ mouse body-part tracking by integrating multi-view video within a global coordinate system, enabling triangulation and bundle adjustment. It introduces a motion-track constraint that enforces temporal consistency of exterior orientation parameters and couples a rigid mouse model with deep-learned deformable body-part movements, implemented via a transformer/LSTM framework to predict deformable coordinates. The approach is instantiated through a functional, stochastic, and initialization-aware least-squares bundle adjustment across three cameras, validated with simulated data and real-world observations from DeepLabCut, demonstrating more complete and geometrically accurate 3D motion estimates for subsequent behavior analysis. The method has practical significance for detailed behavioral studies in mice and potentially other subjects where occlusions limit single-view tracking, providing richer motion representations to support classification and interpretation of learned behaviors.

Abstract

Tracking mouse body parts in video is often incomplete due to occlusions such that - e.g. - subsequent action and behavior analysis is impeded. In this conceptual work, videos from several perspectives are integrated via global exterior camera orientation; body part positions are estimated by 3D triangulation and bundle adjustment. Consistency of overall 3D track reconstruction is achieved by introduction of a 3D mouse model, deep-learned body part movements, and global motion-track smoothness constraint. The resulting 3D body and body part track estimates are substantially more complete than the original single-frame-based body part detection, therefore, allowing improved animal behavior analysis.

Tracking Mouse from Incomplete Body-Part Observations and Deep-Learned Deformable-Mouse Model Motion-Track Constraint for Behavior Analysis

TL;DR

This work tackles the challenge of occlusion-driven gaps in mouse body-part tracking by integrating multi-view video within a global coordinate system, enabling triangulation and bundle adjustment. It introduces a motion-track constraint that enforces temporal consistency of exterior orientation parameters and couples a rigid mouse model with deep-learned deformable body-part movements, implemented via a transformer/LSTM framework to predict deformable coordinates. The approach is instantiated through a functional, stochastic, and initialization-aware least-squares bundle adjustment across three cameras, validated with simulated data and real-world observations from DeepLabCut, demonstrating more complete and geometrically accurate 3D motion estimates for subsequent behavior analysis. The method has practical significance for detailed behavioral studies in mice and potentially other subjects where occlusions limit single-view tracking, providing richer motion representations to support classification and interpretation of learned behaviors.

Abstract

Tracking mouse body parts in video is often incomplete due to occlusions such that - e.g. - subsequent action and behavior analysis is impeded. In this conceptual work, videos from several perspectives are integrated via global exterior camera orientation; body part positions are estimated by 3D triangulation and bundle adjustment. Consistency of overall 3D track reconstruction is achieved by introduction of a 3D mouse model, deep-learned body part movements, and global motion-track smoothness constraint. The resulting 3D body and body part track estimates are substantially more complete than the original single-frame-based body part detection, therefore, allowing improved animal behavior analysis.
Paper Structure (19 sections, 10 equations, 7 figures, 1 table)

This paper contains 19 sections, 10 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Model and Method
  4. Global Co-Ordinate System
  5. Modelling the Relationship Between Body Parts and Camera Orientation
  6. Motion-Track Constraint
  7. Rigid Mouse Model
  8. Deep-Learning Deformable Mouse Model
  9. First Experiment Based on Simulated Data
  10. Least-Squares Adjustment
  11. Functional Model
  12. Stochastic Model
  13. Approximations for Initialization
  14. Data
  15. Simulated Data
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Mouse lock box design and mouse in multi-cam video views solving the lock-box riddle
  • Figure 2: Grid before (blue) and after transform (green) with $\mathbf{H}_t^k {\mathbf{S}_t^k}^{-1}$
  • Figure 3: Rigid mouse model.
  • Figure 4: Tokens input to a deep net. The orthonormal vector triplets indicate the mouse model moving through space, plus ("+") signs indicate rigid body part, and paw "wiggles" indicate deforming body part.
  • Figure 5: Simulated deformable mouse model
  • ...and 2 more figures