Automated behavior recognition for large animals in situ

Scope. Tools that produce behavior labels from imagery, including drone video, camera-trap video, fixed or handheld field video, for large-bodied animals in situ, i.e. non-lab settings. Pose estimation is included only as an intermediate representation feeding behavior classification. On-body biologging (accelerometry, GPS collars) is a different modality answering different questions and is out of scope.

Existing tools for detection and tracking stages are already cataloged and maintained by others, more thoroughly and more currently than we could. Start there:

• Camera Trap ML Survey (Dan Morris) — publicly available ML models for camera traps, plus datasets, tools, and an extensive literature list. The reference for camera-trap detection and species classification, including MegaDetector.
• Computer Vision for Wildlife Aerial Imagery (Benjamin Kellenberger) — models, datasets, and methods for drone, aerial survey, thermal, and satellite imagery.

We list below only the detection and tracking components that specific behavior pipelines depend on. For choosing a detector for your own study system, the two resources above are the better starting point.

Level	Meaning
FA	Fully automated — end-to-end processing with minimal human intervention
AV	Automated with validation — automation plus systematic quality checks
SA	Semi-automated — automation for early stages, human review of uncertain outputs
M	Manual or intensive expert review

Every system name links to its DOI. Code, model weights, and dataset links are in the rightmost column.

---

System	Model	Setting	Species scope	Autonomy	Adapt it when	Code and data
YOLO-Behaviour	YOLOv8, frame-wise	Drone, fixed, handheld	Birds, ungulates	AV	Behaviors are posturally distinct in a single frame and annotation budget is small	code
FERAL	Direct video-to-behavior mapping	Drone, camera trap, lab	Zebras, apes, mice	AV	You have reliable tracks and want to skip pose entirely	code · site
KABR behavior models	X3D, SlowFast, I3D	Drone, in situ	Ungulates	AV	Behaviors are temporally defined and you can supply animal-centered clips	weights · data · analysis
Wild ActionFormer	UniformerV2 + self-supervision	Camera trap, single reserve	11 endangered species	SA	Long-tailed camera-trap behavior data; cross-site generality untested	code
Zero-shot VLM	Vision–language models, no training	Camera trap	Mammal assemblage	SA	You have no labeled data and want a baseline before committing annotation effort	data
DeepEthogram	Optical flow + ResNet + TGM	Fixed camera; lab	Mice, flies	M	Your camera is fixed and a single subject is in frame	code
BehaveAI	Motion-based object detection and behavior classification	Fixed camera; lab and field	Small-bodied taxa (validated on pigeons and microscopic subjects)	SA	Subjects are small and motion-defined against a static background; untested on large animals in situ	code · v1.2.0 · data

System	Model	Setting	Species scope	Autonomy	Adapt it when	Code and data
ASBAR	DeepLabCut → PoseConv3D	Camera trap, in situ	Great apes	SA	Labeled video is scarce and pose is affordable; validate the pose stage first	code
SimBA	Random forest on pose features; explainability layer	Fixed camera; lab	Rodents	M	Listed for completeness; fixed-camera assumption	code
B-SOiD	Unsupervised clustering of pose dynamics	Fixed camera; lab	Rodents	M	Listed for completeness; fixed-camera assumption	code
VAME	Variational RNN on pose	Fixed camera; lab	Rodents	M	Listed for completeness; fixed-camera assumption	code
keypoint-MoSeq	Switching linear dynamical system on keypoints	Fixed camera; lab	Rodents	M	Listed for completeness; fixed-camera assumption	code · jax-moseq · site

On the pose route. Here, we consider pose as an intermediate representation of behavior. Pose classification has been challenging to deploy in the field. ASBAR reports approximately 54% PCK out of domain, and those errors cascade into the behavior classifier. The four laboratory tools above all assume a fixed camera; thus none is recommended for field imagery. We list them here so readers know they were considered. We recommend validating the pose stage before committing to a pose-based pipeline.

System	Model	Setting	Species scope	Autonomy	Adapt it when	Code and data
DeepLabCut	ResNet-50/101; DLCRNet_ms5	Fixed camera; lab, captive	Rodents, primates, fish	M	Your camera is fixed and the arena bounded; expect retraining per setup	code · multi-animal
SLEAP	Configurable U-Net	Fixed camera; lab	Insects, rodents	M	Multi-animal pose in an arena; specialist models, weak transfer	site
DeepPoseKit	Stacked DenseNet	Lab and drone, in situ	Insects, Grévy's zebra	SA	You have reliable detections and crops and need posture rather than behavior labels	code
SuperAnimal	HRNet, ViTPose; Quadruped-80K	Lab and zoo benchmarks	45+ quadrupeds	SA	Your species is a side-view quadruped and you want zero-shot pose before fine-tuning	DLC ModelZoo
DeepWild	DeepLabCut, ape models	Handheld video, in situ forest	Chimpanzee, bonobo	SA	You work on wild apes and can build on released field-trained models	code
AP-10K	HRNet, SimpleBaseline	Naturalistic imagery	54 species	—	You need a pose benchmark including large African mammals	data
APT-36K	HRNet, ViTPose; SwinTrack	Web video	30 species	—	You need joint pose-and-tracking data; no behavior annotations	data

System	Model	Setting	Stages	Autonomy	Adapt it when	Code and data
kabr-tools	YOLO + BoT-SORT → mini-scenes → X3D	Drone and camera trap, in situ	Det + Trk + Beh	AV	You need a modular pipeline whose stages can be replaced or manually corrected	code · data
Price et al. 2023	Smarter-LabelMe + CV pipeline	Drone, in situ Kenya	Det + Trk + Beh	SA	You want to bootstrap a large training set from a small manual one	code
Koger et al. 2023	Detectron2 + SfM + DeepPoseKit	Drone, in situ Kenya, Ethiopia	Det + Trk + Pose	SA	You need georeferenced movement and posture in a 3D landscape model	code
SWIFT / MAROON	Mask R-CNN + mask-guided SlowFast	Camera trap, Germany	Det/Seg + Trk + Beh	SA	You can afford mask annotation and study single-animal actions	—
BEHAVE	AI-assisted detection + manual coding	Video, in situ	Det + manual Beh	SA	You want detection-assisted manual coding rather than automated classification	code · app
SIPEC	Mask R-CNN + IdNet + PoseNet + BehaveNet	Fixed camera; captive	Det + ID + Pose + Beh	M	Your setting is a bounded enclosure with known individuals	code
AlphaChimp	Deformable DETR + Video Swin-L	Fixed camera; zoo	Det + Trk + Beh	M	Your setting closely matches a fixed-camera captive group	project

Upstream components these pipelines rely on. For detector selection, please see the two maintained resources at the top of this page. For tracking, the components that appear in behavior pipelines are BoT-SORT (code), the default in kabr-tools, and SA-FARI (data) as training and evaluation data for general-purpose wild-animal tracking.

Dataset	Modality	Scope	Species and classes	Use it for	Data
KABR	Drone	Mpala, Kenya	3 ungulates, 8 classes	Savanna ungulates observed from above	HuggingFace
BaboonLand	Drone	Mpala, Kenya	Olive baboon, 12 classes	Terrestrial primates from the air	site
PanAf20K	Camera trap	18 African sites	2 ape species, 9 fine classes	Forest camera-trap footage of large primates	site
MammAlps	Multi-view camera trap + audio	Swiss National Park	5 species, 11 activities	Synchronized multi-view or audio-video data	code · data
LoTE-Animal	Camera trap	Sichuan, 12 years	11 endangered species	Long-term camera-trap monitoring	site
CVB	Fixed camera	Australian pasture	Cattle, 11 classes	Livestock behavior in an enclosed paddock	data
MammalNet	Web video	YouTube, captive-biased	173 species, 12 classes	Pretraining only; not field-representative	site

Resource	What it measures
Terra Incognita	How much detection accuracy is lost at camera locations not seen in training
PanAf-FGBG	Whether your behavior model is learning location rather than behavior — background-only models retained ≥70% uAP on wild ape video
LoTE-Animal cross-domain split	Web-to-field domain shift; reported 49–60% performance drop
Horse-10	Whether a pose model generalizes to unseen individuals of the training species

Resource	Relevance
Computer vision for primate behavior analysis in the wild	Obstacles to field deployment; notes the absence of integrated detection–tracking–ID–behavior frameworks
Paradigms and Principles for Integrating Nature Technology Data	Six paradigms for combining sensor streams; where multimodal integration breaks down in practice
FAIR² Drones	Metadata and annotation standard letting one wildlife drone dataset serve ecology, robotics, and computer vision

---

Item	Reason
Detection and tracking catalogs	Maintained better elsewhere — see the two resources at the top
Accelerometry, GPS-collar behavioral inference	Different modality, different measurement methods, different questions
AnimalTrack, Animal Kingdom	Web-scraped; taxonomically far broader than large animals; AnimalTrack has no behavior labels
idtracker.ai, AlphaTracker	Tracking only, closed populations, controlled lighting
Counting-focused aerial work	Detection without behavior; covered by the aerial imagery resource above

Additions and corrections are welcome via pull request. For a new entry please include: a DOI, the model architecture, the setting and species it was validated on (not the settings it claims to support), the pipeline stages it covers, and a link to code or weights where available. Autonomy-level suggestions should say what evidence supports them.

Entries must produce or evaluate behavior labels. Detection- and tracking-only tools belong in the two maintained resources linked at the top.