Suicide Prediction Method Combines AI and Face-to-Face Screening

A new observational study from Vanderbilt University Medical Center points to solutions for efficient clinical prediction of suicide attempt or suicidal thinking in adults.

Reported May 13 in JAMA Open by Drew Wilimitis, Colin Walsh, MD, MA, and colleagues, the study compares an artificial intelligence algorithm with face-to-face screening.

Predictions improved across the board when these two methods were combined, producing a so-called ensemble learning method.

“This test is a rare examination of the synergy between real-time AI and face-to-face screening for estimating suicide risk,” said Walsh, associate professor of Biomedical Informatics, Medicine and Psychiatry and Behavioral Sciences.

Suicide has been on the rise in the U.S. for a generation and is estimated to claim the lives of 14 in 100,000 Americans each year, making it the nation’s 10th leading cause of death.

In 2018, VUMC’s adult emergency department implemented the Columbia Suicide Severity Rating Scale (C-SSRS), with patients routinely answering a six-item questionnaire as part of triage. Meanwhile, teams in other areas of VUMC use this questionnaire for face-to-face screening on an as-needed basis.

In 2019, VUMC on a test basis turned on a machine learning algorithm developed by Walsh and his team, called the Vanderbilt Suicide Attempt and Ideation Likelihood, or VSAIL. The algorithm ran silently in the background, generating risk scores for every patient encounter based on information in the electronic health record. While testing continued, no VSAIL scores were included in health records at VUMC.

The new study focused on 120,398 patient visits over the 15 months concluding in September 2020. For every encounter in which both risk assessment methods were used, health records were reviewed 180 days later for suicide attempts or suicidal thoughts. In all there were 514 documented suicide attempts in the study group (205 occurring within 30 days of a patient visit), and 3,126 cases of suicidal ideation. Suicide deaths were not tracked. The predictive methods were compared with reference to various risk thresholds and patient follow-up periods.

The ensemble method nearly always came out best compared to C-SSRS or VSAIL alone.

“Ensemble learning combined the respective strengths of clinical screening and machine learning, while mitigating their weaknesses,” said Wilimitis, a statistical analyst in Walsh’s lab. “This complementary nature between traditional clinical assessment and statistical prediction should encourage further integration of clinician expertise, patient input and artificial intelligence for effective risk detection.” 

Among the 10% of patients with the highest risk scores per C-SSRS or VSAIL, around one out of 200 went on to attempt suicide within 30 days. But with the ensemble method, around three out of every 200 patients in the top risk decile attempted suicide within 30 days.

“Results of our study show integrating clinician-led screening with algorithms improves suicide attempt prediction — even if those algorithms were never trained with face-to-face screening in the first place,” Walsh said. “This study is a perfect example of the idea that smart humans plus machines are better than either one alone.”

Walsh’s team will next conduct a randomized clinical trial at VUMC of the artificial intelligence algorithm for outpatient clinical decision support. In a random half of visits, high risk scores per VSAIL will automatically trigger an electronic health record alert for clinicians, recommending face-to-face screening for suicide risk. The research team will first of all examine rates of face-to-face screening in cases (high risk per VSAIL flagged for clinicians) and controls (unflagged high risk).

Others on the study include Robert Turer, MD, MS, Michael Ripperger, Allison McCoy, PhD, Sarah Sperry, PhD, Elliot Fielstein, PhD, and Troy Kurz, MD. The study was supported by the National Institutes of Health (MH121455) and the Evelyn Selby Stead Fund for Innovation.