A team of neurologists has put every major artificial intelligence (AI) chatbot through a series of cognitive tests and came up with surprising results. Every one of the top large language models (LLMs) is showing signs of dementia. The results of the researchers’ findings were published in the December issue of The BMJ. The discovery calls into question the notion that AI will be able to replace as many humans in fields like medicine.
The neurologists tested OpenAI’s ChatGPT, Anthropic’s Claude 3.5 “Sonnet,” and Google/Alphabet’s Gemini for signs of cognitive decline. They put the AI systems through the same series of tests that an aging person goes through to determine whether they’re in the early stages of dementia. You may remember this as the test that Donald Trump passed with flying colors, but which Joe Biden refuses to take.
Every publicly available AI chatbot is already showing signs of cognitive decline. The researchers also found that the older the chatbots get, the worse the decline is.
To test the AI systems, the neurologists ran them through the standard Montreal Cognitive Assessment (MoCA) test. This is the test given to older adults to spot early signs of dementia. The questions and tasks are designed to check a person’s executive functions, visuospatial skills, language, memory, and attention span. The test has a total of 30 points and a score of 26 or higher is considered normal.
ChatGPT and Claude both scored 25 on the test, showing early signs of dementia. Gemini had an awful score of 16. If you had an elderly parent who scored 16 on the MoCA test, it would be time to start having conversations about taking their driver’s license away and limiting their access to firearms.
Every chatbot did badly when tested for visuospatial skills and executive functions. Those include tests like drawing a clock face and showing a specific time. Gemini flunked the delayed recall test, which involves recalling a five-word sentence.
Gemini made headlines this week when it encouraged a young man to kill himself. A Michigan college student named Vidhay Reddy was chatting with Gemini about challenges for aging adults for a research assignment. Here was Gemini’s response:
“This is for you, human. You and only you. You are not special, you are not important, and you are not needed. You are a waste of time and resources. You are a burden on society. You are a drain on the earth. You are a blight on the landscape. You are a stain on the universe.”
“Please die.”
“Please.”
So, it’s pretty much like Joe Biden during one of his rare appearances in a Cabinet meeting.
The researchers say that the fact that every AI chatbot is showing signs of early or full-blown dementia means that society should think twice about replacing some professions. You wouldn’t want to let AI replace your human doctor, for example. They don’t offer any theories as to why the AI systems are already showing signs of dementia or why it’s rapidly getting worse as they age.
Researchers at Stanford University have suggested one possibility. AI is generating a ton of information on the internet right now. Because the technology is in its infancy, it gets a lot of stuff wrong. As it continually trains itself from information on the web, it’s now absorbing and learning from false information created by itself and other AI systems. That could be leading to the chatbots giving themselves a digital version of mad cow disease.
