Smartphone app uses AI to detect depression from facial cues

Dartmouth researchers report they have developed the first smartphone application that uses artificial intelligence paired with facial-image processing software to reliably detect the onset of depression before the user even knows something is wrong.

Called MoodCapture, the app uses a phone’s front camera to capture a person’s facial expressions and surroundings during regular use, then evaluates the images for clinical cues associated with depression. In a study of 177 people diagnosed with major depressive disorder, the app correctly identified early symptoms of depression with 75% accuracy.

These results suggest the technology could be publicly available within the next five years with further development, said the researchers, who are based in Dartmouth’s Department of Computer Science and Geisel School of Medicine. The team published their paper on the arXiv preprint database in advance of presenting it at the Association of Computing Machinery’s CHI 2024 conference in May. Papers presented at CHI are peer-reviewed prior to acceptance and will be published in the conference proceedings.

“This is the first time that natural ‘in-the-wild’ images have been used to predict depression,” said Andrew Campbell, the paper’s corresponding author and Dartmouth’s Albert Bradley 1915 Third Century Professor of Computer Science. “There’s been a movement for digital mental-health technology to ultimately come up with a tool that can predict mood in people diagnosed with major depression in a reliable and non-intrusive way.”

“People use facial recognition software to unlock their phones hundreds of times a day,” said Campbell, whose phone recently showed he had done so more than 800 times in one week.

“MoodCapture uses a similar technology pipeline of facial recognition technology with deep learning and AI hardware, so there is terrific potential to scale up this technology without any additional input or burden on the user,” he said. “A person just unlocks their phone and MoodCapture knows their depression dynamics and can suggest they seek help.”

For the study, the application captured 125,000 images of participants over the course of 90 days. People in the study consented to having their photos taken via their phone’s front camera but did not know when it was happening.

A first group of participants was used to program MoodCapture to recognize depression. They were photographed in random bursts using the phone’s front-facing camera as they answered the question, “I have felt down, depressed, or hopeless.” The question is from the eight-point Patient Health Questionnaire or PHQ-8, which is used by clinicians to detect and monitor major depression.

The researchers used image-analysis AI on these photos so that MoodCapture’s predictive model could learn to correlate self-reports of feeling depressed with specific facial expressions—such as gaze, eye movement, positioning of the head, and muscle rigidity—and environmental features such as dominant colors, lighting, photo locations, and the number of people in the image.

The concept is that every time a user unlocks their phone, MoodCapture analyzes a sequence of images in real-time. The AI model draws connections between expressions and background details found to be important in predicting the severity of depression, such as eye gaze, changes in facial expression, and a person’s surroundings. 

Over time, MoodCapture identifies image features specific to the user. For example, if someone consistently appears with a flat expression in a dimly lit room for an extended period, the AI model might infer that person is experiencing the onset of depression.

The researchers tested the predictive model by having a separate group of participants answer the same PHQ-8 question while MoodCapture photographed them and analyzed their photos for indicators of depression based on the data collected from the first group. It is this second group that the MoodCapture AI correctly determined were depressed or not with 75% accuracy. 

“This demonstrates a path toward a powerful tool for evaluating a person’s mood in a passive way and using the data as a basis for therapeutic intervention,” said Campbell, noting that an accuracy of 90% would be the threshold of a viable sensor. “My feeling is that technology such as this could be available to the public within five years. We’ve shown that this is doable.”

MoodCapture meets major depression on the irregular timescale on which it occurs, said Nicholas Jacobson, a study co-author and assistant professor of biomedical data science and psychiatry in Dartmouth’s Center for Technology and Behavioral Health.

“Many of our therapeutic interventions for depression are centered around longer stretches of time, but these folks experience ebbs and flows in their condition. Traditional assessments miss most of what depression is,” said Jacobson, who directs the AI and Mental Health: Innovation in Technology Guided Healthcare (AIM HIGH) Laboratory.

“Our goal is to capture the changes in symptoms that people with depression experience in their daily lives,” Jacobson said. “If we can use this to predict and understand the rapid changes in depression symptoms, we can ultimately head them off and treat them. The more in the moment we can be, the less profound the impact of depression will be.” 

Jacobson anticipates that technologies such as MoodCapture could help close the significant gap between when people with depression need intervention and the access they have to mental-health resources. On average, less than 1% of a person’s life is spent with a clinician such as a psychiatrist, he said. “The goal of these technologies is to provide more real-time support without adding an additional pressure on the care system,” Jacobson said.

An AI application like MoodCapture would ideally suggest preventive measures such as going outside or checking in with a friend instead of explicitly informing a person they may be entering a state of depression, Jacobson said.

“Telling someone something bad is going on with them has the potential to make things worse,” he said. “We think that MoodCapture opens the door to assessment tools that would help detect depression in the moments before it gets worse. These applications should be paired with interventions that actively try to disrupt depression before it expands and evolves. A little over a decade ago, this type of work would have been unimaginable.”

The study stems from a National Institutes of Mental Health grant Jacobson leads that is investigating the use of deep learning and passive data collection to detect depression symptoms in real-time. It also builds off a 2012 study led by Campbell’s lab that collected passive and automatic data from the phones of participants at Dartmouth to assess their mental health.

But the advancement of smartphone cameras since then allowed the researchers to clearly capture the kind of “passive” photos that would be taken during normal phone usage, Campbell said. Campbell is director of emerging technologies and data analytics in the Center for Technology and Behavioral Health where he leads the team developing mobile sensors that can track metrics such as emotional state and job performance based on passive data.

The new study shows that passive photos are key to successful mobile-based therapeutic tools, Campbell said. They capture mood more accurately and frequently than user-generated photographs—or selfies—and do not deter users by requiring active engagement. “These neutral photos are very much like seeing someone in-the-moment when they’re not putting on a veneer, which enhanced the performance of our facial-expression predictive model,” Campbell said. 

Subigya Nepal, a Guarini School of Graduate and Advanced Studies PhD candidate in Campbell’s research group who, along with PhD student Arvind Pillai, Guarini, is co-lead author of the study, said the next steps for MoodCapture include training the AI on a greater diversity of participants, improving its diagnostic ability, and reinforcing privacy measures.

The researchers envision an iteration of MoodCapture for which photos never leave a person’s phone, Nepal said. Pictures would instead be processed on a user’s device to extract facial expressions associated with depression and convert them into code for the AI model. “Even if the data ever does leave the device, there would be no way to put it back together into an image that identifies the user,” he said.

Meanwhile, the application’s accuracy could be enhanced on the consumer end if the AI is designed to expand its knowledge based on the facial expressions of the specific person using it, Nepal said. 

“You wouldn’t need to start from scratch—we know the general model is 75% accurate, so a specific person’s data could be used to fine-tune the model. Devices within the next few years should easily be able to handle this,” Nepal said. “We know that facial expressions are indicative of emotional state. Our study is a proof of concept that when it comes to using technology to evaluate mental health, they’re one of the most important signals we can get.”