PHILADELPHIA— An automated, online triage tool developed by Penn Medicine categorized nearly every one of the patients who used it into a safe severity level, a new study shows. Published today in Applied Clinical Informatics, the study analyzing the COVID-19 Triage Tool found that just six patients of the 782 analyzed had symptoms that were more severe than what the system assessed. But even in those cases, clinicians working alongside the system were able to upgrade the patients’ assessment to the proper level of severity and attention.
“The triage tool was incredibly effective at safely categorizing patients to their appropriate triage level or a slightly higher acuity,” said the study’s co-lead author, Elana Meer, an MD/MBA student in the Perelman School of Medicine at the University of Pennsylvania and the Wharton School. “We aimed to describe the process of developing and testing an automated, publicly available COVID-19 symptom triage tool that could provide a roadmap for others. Our primary priority in this work was to evaluate safety and efficacy and, to that end, I’d say that this was an incredibly successful system that helped both patients and providers at an extremely trying time.”
When COVID-19 struck the United States, it stretched thin the availability of doctors and nurses who typically assess patients. On top of that, social distancing complicated those assessments. In response, a team at Penn Medicine leaned on something patients have already been doing for years: Checking their symptoms online.
The COVID-19 Triage Tool was deployed May 4, 2020, amid the first large surge of the virus on the East Coast. The tool consisted of a frequently asked questions (FAQ) page and an automated, online chatbot that asked patients questions through their browser or patient portal. Unlike common and commercially available symptom checkers, the triage tool was built by Penn Medicine and integrated within its systems so that patients could be guided to the right level of care. Often, online symptom checkers get to a point where they direct patients to call their primary care doctor, and then the doctor has to repeat the evaluation. This system was set up to do the early legwork, document it, and efficiently send patients to the right people without doubling up on human workloads.
“When developed thoughtfully, symptom checkers should both improve patient experience and take some of the load off care teams,” said study co-author Krisda Chaiyachati, MD, the medical director of Penn Medicine OnDemand, the health system’s virtual visit service. “A proper triage tool, especially one during a crisis like COVID-19, should improve patients’ experience by enabling 24/7, on-demand self-service while also safely offloading patient call centers, freeing up clinicians staffing them to attend to patient calls that more require their expertise and judgment.”
The researchers assessed the COVID-19 Triage Tool from its launch until April 2021, during which time 20,930 patients used it. For the study, 782 patients were analyzed since they had been categorized by the tool and had also contacted the health system’s formal triage phone line. That way, the tool’s categorization could be directly compared to a live clinician’s assessment.
Roughly half of the patients who used the tool were judged by it to be asymptomatic, while more than 30 percent were categorized as either moderately or severely sick. Those patients were all directed to immediately call their physician or 911, respectively. The tool did not assess whether a patient “had” COVID-19, just how concerning their symptoms were and what type of care they were likely to need.
To ensure that patients were being safely handled, the COVID-19 Triage Tool was designed to be conservative in its assessments. As such, a little over 29 percent of the assessments the tool made matched exactly how the clinicians working alongside it would have classified patients’ status. But 70 percent of the patients were assessed at a higher severity than clinicians would have categorized them. That meant that the number of patients being triaged by the tool to a lower severity than what they actually were was miniscule.
“From our standpoint, this was a win, given that 99.3 percent of patients either were assessed exactly as an experienced clinician assessed them or were referred – out of an abundance of caution – to a clinician who later judged these patients to be lower risk than had been originally been suggested,” said co-lead author Maguire Herriman, an MD/MBA student in the Perelman School of Medicine at the University of Pennsylvania and the Wharton School. “Later research can assess absolute accuracy and precision for this kind of tool. It safely served its purpose for us in this situation.”
Even if physicians were reconsidering the patients classified at a higher severity than needed, much of the work had already been done for them. The clinician answering the phone for patients referred to the clinician hotline could understand at a glance the questions and answers that the triage tool had already reviewed. And it was presented in a useful way that saved time.
While the COVID Clinical Triage Tool was discontinued in May 2021 when its use dropped significantly, it filled an important role at a critical time and provides a blueprint for future solutions.
There will be some need for further refining the product for those future situations. Only about 13 percent of the users were patients over 61 years old, which might call for adjustments to the system to make it more friendly to less tech-savvy populations. That could include phone interviews or other interventions.
“Future work might identify different ways of promoting tools like this with older adults and consider offering accessible, readily-understood, coaching sessions to those individuals interested in using symptom checker tools,” said senior author John D. McGreevey III, MD, an associate chief medical information officer at the University of Pennsylvania Health System. “By taking such steps, whether with older adults or other population subgroups, we can assure that a wide range of users become comfortable and confident in using symptom checker tools.”
###
Penn Medicine is one of the world’s leading academic medical centers, dedicated to the related missions of medical education, biomedical research, and excellence in patient care. Penn Medicine consists of the Raymond and Ruth Perelman School of Medicine at the University of Pennsylvania (founded in 1765 as the nation’s first medical school) and the University of Pennsylvania Health System, which together form a $8.9 billion enterprise.
The Perelman School of Medicine has been ranked among the top medical schools in the United States for more than 20 years, according to U.S. News & World Report’s survey of research-oriented medical schools. The School is consistently among the nation’s top recipients of funding from the National Institutes of Health, with $496 million awarded in the 2020 fiscal year.
The University of Pennsylvania Health System’s patient care facilities include: the Hospital of the University of Pennsylvania and Penn Presbyterian Medical Center—which are recognized as one of the nation’s top “Honor Roll” hospitals by U.S. News & World Report—Chester County Hospital; Lancaster General Health; Penn Medicine Princeton Health; and Pennsylvania Hospital, the nation’s first hospital, founded in 1751. Additional facilities and enterprises include Good Shepherd Penn Partners, Penn Medicine at Home, Lancaster Behavioral Health Hospital, and Princeton House Behavioral Health, among others.
Penn Medicine is powered by a talented and dedicated workforce of more than 44,000 people. The organization also has alliances with top community health systems across both Southeastern Pennsylvania and Southern New Jersey, creating more options for patients no matter where they live.
Penn Medicine is committed to improving lives and health through a variety of community-based programs and activities. In fiscal year 2020, Penn Medicine provided more than $563 million to benefit our community.
