Algorithm that analyzes patients' speech, facial expressions holds promise for diagnosing mental illness, study finds

Certain psychiatric diagnoses and their symptoms can be identified in patients when analyzing their speech and facial movements through machine learning technology, a recent study found

The study results suggest that analysis of individuals’ speech and facial movement holds significant promise for creating a more objective approach to psychiatric diagnosis and treatment.

The study was led by Michael Birnbaum, M.D., an assistant professor at the Feinstein Institutes for Medical Research and director of the Early Treatment Program at Northwell Health. It aimed to determine whether reliable inferences can be extracted from audio-visual patterns in patients with schizophrenia spectrum disorders and bipolar disorders, which are accompanied by established speech and facial expression changes. 

To capture the audio-visual data, Birnbaum and his team recorded 146 total evaluation interviews with 89 participants. They used publicly available software to turn the videos and audio into data points that could be analyzed. Birnbaum's team then partnered with IBM analysts who built algorithms to analyze the data.

The algorithms successfully differentiated between schizophrenia spectrum disorders and bipolar disorder in participants and identified various symptoms with high degrees of accuracy, according to the study. Symptoms included blunted affect, avolition, lack of vocal inflection, asociality and worthlessness. Researchers generally found that facial features, such as use of the cheek-raising and chin-raising muscles, were the strongest predictors for men and voice features were the most common indicators for women.

Compared to the rest of medicine, “psychiatry is somewhat antiquated," Birnbaum told Fierce Healthcare. Conditions are self-reported, which can be limiting or even be flawed. There is little way to get objective lab values comparable to a clinician using X-rays or blood tests to make a diagnosis.

RELATED: The money behind mental health: How the pandemic increased innovation, investment in behavioral health care

While diagnoses are important, symptoms are what define successful treatment, Birnbaum said. The diagnosis may not go away, but symptoms fluctuate. “That’s how we track progress,” he said. Symptom identification is also important for cases where a patient may not disclose with accuracy their own symptoms, risking treatment efficacy, Birnbaum said. If this technology can be properly incorporated into the clinical workflow, it offers the potential to assess patients more frequently and in a more granular way over time. 

While the technology is still in stages too early to be used in a clinically meaningful setting, Birnbaum acknowledged, the goal is to have “some sort of mental health X-ray.”

“Psychiatry is desperate for objective, clinically meaningful tools,” Birnbaum said, stressing that the technology is not meant to replace the work clinicians do. “We need to improve the way that we collect data.” 

The technology could establish a “baseline digital fingerprint” that then could be used over the course of treatment to monitor a patient’s progress, according to Birnbaum.

Birnbaum sees the next step as testing the technology in clinical workflows and seeing whether it improves outcomes. “We need to make sure we’re delivering that promise,” he said. From a technology standpoint, in Birnbaum’s view, implementation would not be complicated—a clinician can record patients during sessions that are happening anyway, and the software used to generate the data is publicly available.

The real challenge would be integrating the technology with an electronic medical record and addressing reimbursement and privacy concerns, though Birnbaum believes those are all highly addressable.