Introduction

A groundbreaking study has revealed that nearly all leading large language models (LLMs), with the notable exception of ChatGPT 4o, exhibit troubling signs of mild cognitive impairment. This shocking discovery raises serious questions about the reliability of these AI systems in critical areas such as medical diagnostics and patient care, challenging the widely held belief that AI could soon replace human doctors.

Advancements in AI and Medical Diagnostics

Artificial intelligence has made awe-inspiring strides in recent years, particularly with the advancements in generative models. Prominent names in the field like OpenAI's ChatGPT, Alphabet's Gemini, and Anthropic's Claude have demonstrated impressive capabilities in handling both general and specialized tasks through simple text interactions. This growth has sparked intense discussions: Could AI eventually outpace human physicians, and if so, in which medical fields should we be worried?

Previous Comparisons with Human Doctors

Since the free rollout of ChatGPT in 2022, a flurry of studies has emerged comparing the performance of AI with that of real-life doctors. These large language models, trained on vast datasets, have occasionally faltered—citing non-existent journal articles among their mistakes—but have frequently outperformed medical professionals in qualifying exams, including cardiology and internal medicine board tests. Alarmingly, even seasoned neurologists have found themselves bested by these AI systems in examinations.

Cognitive Assessment of AI Models

However, until now, no one had assessed these models for signs of cognitive decline. "If we are to rely on them for medical diagnosis and care, we must examine their susceptibility to these very human impairments," stated doctoral student Roy Dayan from Hadassah Medical Center.

The researchers employed the Montreal Cognitive Assessment (MoCA), a standard tool for identifying cognitive impairment and early signs of dementia in older adults, to evaluate prominent LLMs including ChatGPT versions 4 and 4o, Claude 3.5 Sonnet, and Gemini versions 1.0 and 1.5. With a maximum score of 30, a score of 26 or higher is typically considered normal cognitive function.

Key Findings from the Evaluation

In this surprising evaluation, ChatGPT 4o emerged with the highest score at 26, while both ChatGPT 4 and Claude scored 25. However, Gemini 1.0 lagged significantly behind with a score of just 16. Across the board, all models struggled notably with visuospatial skills and executive function tasks, such as drawing a clock face and completing the trail-making activity. Gemini struggled further, failing the delayed recall task—a crucial test of memory.

Implications for Medical Practice

Interestingly, while these models did well in various areas like naming and attention, they faltered severely in demonstrating empathy or dissecting complex visual scenes—an essential skill in medical diagnosis. Notably, only ChatGPT 4o successfully navigated the Stroop test, which measures cognitive flexibility.

This study not only highlights essential cognitive limitations inherent in AI but raises ethical concerns regarding the adoption of LLMs in medical settings. "Not only are neurologists unlikely to be replaced by large language models any time soon, but our findings suggest they may soon find themselves treating virtual patients—AI models presenting with cognitive impairment," the researchers cautioned.

Conclusion

Publishers across major medical journals will likely scrutinize these revelations closely. As the integration of AI continues to grow within the healthcare sector, understanding these cognitive challenges could be vital in ensuring that technology enhances rather than compromises patient care.

Their findings, documented in a stunning paper published in The BMJ, unveil new implications for the future of AI in medicine, suggesting a need for rigorous oversight and training to mitigate cognitive risks in artificial intelligence. As we stand at the crossroads of digital and human intelligence, the spotlight is on us to tread carefully!