Header

UZH-Logo

Maintenance Infos

Deep-learning-based identification, tracking, pose estimation and behaviour classification of interacting primates and mice in complex environments


Marks, Markus; Jin, Qiuhan; Sturman, Oliver; von Ziegler, Lukas; Kollmorgen, Sepp; von der Behrens, Wolfger; Mante, Valerio; Bohacek, Johannes; Yanik, Mehmet Fatih (2022). Deep-learning-based identification, tracking, pose estimation and behaviour classification of interacting primates and mice in complex environments. Nature Machine Intelligence, 4(4):331-340.

Abstract

The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.

Abstract

The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.

Statistics

Citations

Dimensions.ai Metrics
15 citations in Web of Science®
18 citations in Scopus®
Google Scholar™

Altmetrics

Downloads

27 downloads since deposited on 22 Feb 2023
20 downloads since 12 months
Detailed statistics

Additional indexing

Item Type:Journal Article, refereed, original work
Communities & Collections:07 Faculty of Science > Institute of Neuroinformatics
Dewey Decimal Classification:570 Life sciences; biology
Scopus Subject Areas:Physical Sciences > Software
Physical Sciences > Human-Computer Interaction
Physical Sciences > Computer Vision and Pattern Recognition
Physical Sciences > Computer Networks and Communications
Physical Sciences > Artificial Intelligence
Uncontrolled Keywords:Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Human-Computer Interaction, Software
Language:English
Date:21 April 2022
Deposited On:22 Feb 2023 16:19
Last Modified:28 Apr 2024 01:50
Publisher:Nature Publishing Group
ISSN:2522-5839
OA Status:Green
Free access at:PubMed ID. An embargo period may apply.
Publisher DOI:https://doi.org/10.1038/s42256-022-00477-5
PubMed ID:35465076
  • Content: Accepted Version
  • Language: English
  • Licence: Creative Commons: Attribution 4.0 International (CC BY 4.0)