NTU & Harvard Reveal AI Limits in Care

Artificial intelligence chatbots have become increasingly popular in health information access, raising the question of whether they can meaningfully support stroke prevention and care. Stroke remains one of the leading causes of death and disability worldwide, disproportionately affecting socially disadvantaged populations.

A recent study led by National Taiwan University's (NTU) College of Public Health in collaboration with the Harvard T.H. Chan School of Public Health evaluated three large language models:ChatGPT, Claude, and Gemini, across four stages of stroke care: prevention, diagnosis, treatment, and rehabilitation. Using different prompting methods (Zero-shot Learning, Chain of Thought, and Talking Out Your Thoughts), clinical experts assessed accuracy, hallucinations, specificity, empathy, and actionability, referencing the minimum passing standard of a medical licensing exam. The findings, published in npj Digital Medicine, show that while each prompting method had strengths, overall model performance remained clinically insufficient and inconsistent across scenarios.

The research team emphasized that although generative AI may help alleviate health inequalities and workforce shortages, especially in resource-limited regions, these tools currently fall short of professional medical standards. Safer application will require continued model improvement, stronger user education on effective prompting, rigorous clinical validation, regulation, and oversight by medical professionals.