Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expression

Paimon Goulart, Jordan Steinhauser, Kylene Shuler, Edward Korzus, Jia Chen, Evangelos E. Papalexakis

TL;DR

The paper addresses the need for scalable, fine-grained annotations of mouse fear expression aligned with neural data. It repurposes an open-source vision-language model (Qwen2.5-VL) using prompt engineering, in-context learning, and frame-level preprocessing to generate per-second behavioral vectors without model fine-tuning. Results show that each component—prompt design, ICL, and frame-wise processing—contributes to improved labeling, with their combination yielding robust performance across behaviors, including rare freezing and fleeing. This approach enables integrated, multi-modal datasets for studying threat and safety learning, with broad implications for behavioral neuroscience and cross-disciplinary research.

Abstract

Integration of diverse data will be a pivotal step towards improving scientific explorations in many disciplines. This work establishes a vision-language model (VLM) that encodes videos with text input in order to classify various behaviors of a mouse existing in and engaging with their environment. Importantly, this model produces a behavioral vector over time for each subject and for each session the subject undergoes. The output is a valuable dataset that few programs are able to produce with as high accuracy and with minimal user input. Specifically, we use the open-source Qwen2.5-VL model and enhance its performance through prompts, in-context learning (ICL) with labeled examples, and frame-level preprocessing. We found that each of these methods contributes to improved classification, and that combining them results in strong F1 scores across all behaviors, including rare classes like freezing and fleeing, without any model fine-tuning. Overall, this model will support interdisciplinary researchers studying mouse behavior by enabling them to integrate diverse behavioral features, measured across multiple time points and environments, into a comprehensive dataset that can address complex research questions.

Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expression

TL;DR

The paper addresses the need for scalable, fine-grained annotations of mouse fear expression aligned with neural data. It repurposes an open-source vision-language model (Qwen2.5-VL) using prompt engineering, in-context learning, and frame-level preprocessing to generate per-second behavioral vectors without model fine-tuning. Results show that each component—prompt design, ICL, and frame-wise processing—contributes to improved labeling, with their combination yielding robust performance across behaviors, including rare freezing and fleeing. This approach enables integrated, multi-modal datasets for studying threat and safety learning, with broad implications for behavioral neuroscience and cross-disciplinary research.

Abstract

Integration of diverse data will be a pivotal step towards improving scientific explorations in many disciplines. This work establishes a vision-language model (VLM) that encodes videos with text input in order to classify various behaviors of a mouse existing in and engaging with their environment. Importantly, this model produces a behavioral vector over time for each subject and for each session the subject undergoes. The output is a valuable dataset that few programs are able to produce with as high accuracy and with minimal user input. Specifically, we use the open-source Qwen2.5-VL model and enhance its performance through prompts, in-context learning (ICL) with labeled examples, and frame-level preprocessing. We found that each of these methods contributes to improved classification, and that combining them results in strong F1 scores across all behaviors, including rare classes like freezing and fleeing, without any model fine-tuning. Overall, this model will support interdisciplinary researchers studying mouse behavior by enabling them to integrate diverse behavioral features, measured across multiple time points and environments, into a comprehensive dataset that can address complex research questions.
Paper Structure (9 sections, 4 figures)

This paper contains 9 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Data Description
  4. Vision Language Model
  5. Video Preprocessing
  6. In-Context Learning
  7. Labeling Pipeline
  8. Experiments
  9. Conclusions

Figures (4)

  • Figure 1: Overview of our video labeling pipeline.
  • Figure 2: Simple ICL prompt: the model receives a few labeled examples and a target video segment, and is asked to choose one behavior label.
  • Figure 3: Complex ICL prompt: the model is given definitions for each behavior alongside example demonstrations. Note that non-ICL uses the same prompt without examples.
  • Figure 4: Per class F1 scores across all pipeline configurations. We compare the effects of using simple vs. complex prompts, with and without ICL, as well as the impact of full video input vs. frame-wise input. ICL and frame splitting significantly improve performance, especially for rare behaviors.