Study reveals design flaws of chatbot-based symptom-checker apps

UNIVERSITY PARK, Pa. — Millions of people turn to their mobile devices when seeking medical advice. They’re able to share their symptoms and receive potential diagnoses through chatbot-based symptom-checker (CSC) apps.

But how do these apps compare to a trip to the doctor’s office?

Not well, according to a new study. Researchers from Penn State’s College of Information Sciences and Technology have found that existing CSC apps lack the functions to support the full diagnostic process of a traditional visit to a medical facility. Rather, they said, the apps can only support five processes of an actual exam: establishing a patient history, evaluating symptoms, giving an initial diagnosis, ordering further diagnostic tests, and providing referrals or other follow-up treatments.

“These apps do not support conducting physical exams, providing a final diagnosis, and performing and analyzing test results, because these three processes are difficult to realize using mobile apps,” said Yue You, a graduate student in the College of Information Sciences and Technology and lead author on the study.

In the study, the researchers investigated the functionalities of popular CSC apps through a feature review, then examined user experiences by analyzing user reviews and conducting user interviews. Through their user experience analysis, You and her team also found that users perceive CSC apps to lack support for a comprehensive medical history, flexible symptom input, comprehensible questions, and diverse diseases and user groups.

The findings could inform functional and conversational design updates for health care chatbots, such as improving the functions that enable users to input their symptoms or using comprehensible language and providing explanations during conversations.

“Especially in health and medicine, [another question is] is there something else we should consider in the chatbot design, such as how should we let users describe their symptoms when interacting with the chatbot?” said You.

Additionally, the findings could help individuals understand the influence of AI technology, such as how AI could influence or change traditional medical visits.

“In the past, people generally trusted doctors,” You said. “But now with the emergence of AI symptom checkers and the internet, people have more sources of information. How would this information challenge doctors? Do people trust this information and why? I think this work is a starting point to think about the influence of AI symptom checkers.”

###

The findings will be presented at the American Medical Informatics Association (AMIA) Virtual Annual Symposium in November.

You’s work serves as a preliminary study for future in-depth exploration. Currently, she is working to investigate how to design a better, explainable COVID-19 symptom checker with College of Information Sciences and Technology faculty members Xinning Gui, assistant professor; Jack Carroll, Distinguished Professor of Information Sciences and Technology; Yubo Kou, assistant professor; and Chun-Hua Tsai, assistant research professor.