Mapping Behavior with Machine Learning

The subjects in the research videos created by Timothy Dunn, an assistant professor of biomedical engineering at Duke University, aren’t immediately obvious. Rather than show tufts of fur or swishing tails, the animal models––usually mice or rats––are instead depicted by straight lines and colorful dots that move around an otherwise empty screen.

These videos are created through the program DANNCE, short for 3-Dimensional Aligned Neural Network for Computational Ethology, a tool Dunn and his team developed in 2021. Using videos of freely moving rats, the team trained machine-learning algorithms and neural networks to identify and map the precise 3D locations of the body joints on the animals. Researchers could then relate these measurements to data collected from brain recording technologies to examine links between neuronal activity and specific behaviors.

Outside of tools like DANNCE, which raised $2.6 million in pre-seed funding to pursue commercially in late 2024, the methods to track behavior in animal models are still primitive. Researchers will often need to manually watch and score specific behaviors, like sitting or walking, or use simple imaging and computational approaches to measure the position of the animal over time. But these methods are far from accurate, and they haven’t kept pace with the sophisticated neural engineering tools that can image, track and control neural activity in the brain.

“The principal output of the brain is movement and behavior, but we previously haven’t had any tools to precisely track that output in freely moving animals and relate it to underlying neuronal activity,” said Dunn. “If you can’t quantify behavior precisely and comprehensively, you’re not going to get an accurate picture of how disease states or therapeutics affect behavior and movement.”

According to Dunn, this knowledge gap has made it even more difficult to crack the mysteries of devastating motor disorders, like Parkinson’s disease, or neuropsychiatric disorders, like autism spectrum disorders or schizophrenia, that affect behavior.

“Many areas of neuroscience have been hamstrung by the lack of precise, objective, and reproducible descriptions of social behaviors, and DANNCE helps provide a solution to this longstanding problem,” said Dunn. “My team is excited to use tools like machine learning and neural networks to help learn more about these disorders.”

From Coding to the Clinic

Although Dunn’s primary collaborators are neuroscientists, engineers and clinicians at the Duke University Medical Center, he wasn’t always interested in studying biomedical problems.

“I loved computers and engineering, but I also didn’t want to be in a dark basement programming for the rest of my life,” he said. “I’d developed an interest in biology in high school, and after a year of undergrad I decided to explore more of the biology curriculum to see if it was something I could actually pursue.”

As an undergraduate at the University of California, Berkeley, Dunn took courses in neuroscience, neurobiology, neurochemistry and pharmacology and joined a neuroscience lab, which led him to switch his major from electrical engineering and computer science to biology, with specialties in neuroscience and biophysics. While he credits this change for inspiring him to pursue research, he also remembers that he didn’t feel confident bench-top work was his passion either.

“I was young and doubting my trajectory,” he explained. So, like so many uncertain students before him, Dunn started applying to graduate schools.

“I ended up getting accepted at Harvard, where I did a rotation in a lab with zebrafish and whole organism behavior rather than looking at individual neuron channels, which was all I’d up to that point,” said Dunn. “I can trace a lot of my work and interests today back to that lab and that experience. I was writing software, we were using high-speed cameras to collect real-time measurements of the animal’s movements and using deep learning to relate them to neural activity. It was exactly the work I wanted to be doing.”

But it wasn’t until Dunn was hired as an independent research fellow at Duke Forge, a center for actionable health data science that was eventually absorbed into Duke AI Health, that he had a chance to better explore the clinical possibilities of applying machine learning and neural networks to problems in healthcare.

One of his first projects involved working with researchers in the department of anesthesiology at DUMC to track controlled substance theft, also called drug diversion, of prescription drugs like opioids and other pain-relief substances for illegal use or sale. Described as a ‘hidden epidemic,’ in hospitals across the country, this theft can often harm patients and increase the risk of overdoses. Using his computational tools, Dunn and his collaborators analyzed the records of clinicians and staff who had accessed the controlled substance drawers to map out patterns in behavior and relate them to standard drug protocols.

“We were trying to predict strange or aberrant patterns of people grabbing dosage amounts that were abnormal for a given patient’s condition or the type of surgery,” explained Dunn. “The result was an effective drug surveillance system, which we now have a patent on.”

The experience with Duke Forge also enabled Dunn to form his first collaborations with neurosurgeons. Together they were able to use historical datasets from patients to begin making predictive models that could help guide treatment decisions for traumatic brain injuries.

“We were particularly interested in applying this approach to resource limited settings to more effectively triage patients and get the attention of available neurosurgeons,” said Dunn.

Behavior as a New Biomarker

According to Dunn, the development of DANNCE marks a convergence of his clinical work to identify patterns using big data analytics and his computational neuroscience work with animal models.

“We are still developing tools to do basic computational neuroscience, but DANNCE gave us a way to apply our work to solve problems for humans in the clinic, especially those with movement disorders,” he said. “If we could use movement as an objective and sensitive biomarker for a particular disorder or stage of that disorder, we could ideally make better decisions about paths for treatment.”

In people with Parkinson’s disease, for example, patients could visit the clinic and wear a monitor paired with DANNCE to precisely track movements and determine the severity of the disease. If treatments are available, the same technique could be used to track patterns in a patient’s movements for any improvements.

But movement doesn’t always happen in isolation. Dunn and his team have continued to improve their technologies to address other facets of behavior––specifically how animals act in social environments.

In a 2025 paper published in Cell, the team introduced social-DANNCE, or s-DANNCE. Building off the approach they pioneered in DANNCE, the team recorded videos of groups of two to three rats freely interacting in a controlled recording space. These videos were analyzed by a neural network, which was trained to track the movements of the individual animals. By mapping these movements into 3D models of the animal’s joints, the researchers could identify recurring types of movements, which allowed them to sort and classify individual behaviors, like grooming, and social interactions, like chasing, sniffing or fighting.

“Our work with s-DANNCE shows that rat interactions can be separated into hundreds of different social behaviors that can be expressed at different levels,” said Dunn. “Once these interactions are identified, we have new quantitative units that we can use to describe how social interactions change during models of behavioral disorders, like different forms of autism, or when testing drugs.”

The amount of data Dunn and his team have been able to generate from these experiments proves that quantifying movement is an unexplored frontier in the world of neuroscience. From their s-DANNCE paper alone they were able to share a data set of more than 150 million 3D behavioral samples.

“Whether it’s analyzing clinical data with machine learning, our computational movement analysis work, or our collaborations to study movement disorders in humans, we’re constantly pursuing projects that can help catalyze neurobiological discovery,” said Dunn.