Skip to contentSkip to site navigation
Computer Science
Completed Project

Mouse Watching: Utilizing a Multimodal Sensor to Track Multiple Mice for Social Behavior Analysis

Jarrett Holtz, Vassar College ’15 and Profs. Eric Aaron and Bojana Zupan

Mice are a commonly used model for human conditions in biomedical research, specifically for conditions  identified by social interaction impairments, such as autism. To improve the state of the art for mouse observation we utilize the Microsoft Kinect, a multimodal sensor with both depth and color cameras, to develop a prototype for tracking multiple mice and to obtain more information than was previously possible at a lower cost to researchers. Current methods include radio frequency tracking which could discern positional information about multiple mice, but no information beyond proximity. Other methods use low fps black and white cameras which follow one to two mice, without reliable information about their orientation, and in limited scenarios. By using the  depth and color sensors we avoid some of the difficulties involved with computer vision, and can then follow multiple mice. In order to remove camera noise and locate mice our method utilizes a novel domain specific noise filtering method, along with a system for background recognition and updating, to isolate mice in the depth frame. This allows us to use cluster analysis to assign all pixels belonging to a mouse to a cluster, and information about directions of movement is then used to identify both position and orientation of each mouse. Using areas identified by the depth camera we specify pixels to examine in the color frame to obtain information available only to the color camera, such as unique identifiers for mice, making it possible to attribute actions to specific mice. Our system is currently capable of tracking multiple mice and their orientation at a frame rate of ~20fps, an improvement on tracking one mouse at ~7fps. By making it possible to track multiple mice this enables researchers to obtain information about social interactions in groups of mice in an automated way. Ongoing work will see development of more robust unique identification of mice, and domain specific analysis of their interactions, along with fine tuning to improve speed, precision, and the interface.