As writer Ian Bogost pointed out in The Atlantic, there are many reasons why ChatGPT sometimes struggles.

But first and foremost, it lacks the ability to truly understand the complexity of human language and conversation. Chatbots are trained to generate words based on a given input, but they do not have the ability to truly comprehend the meaning behind those words.

This means that the responses they put together can be shallow and lacking in depth and insight.

John P. Nelson, who is a Postdoctoral Research Fellow in Ethics and Societal Implications of Artificial Intelligence at Georgia Institute of Technology, agrees that so far, large language models, for all their complexity, "are actually really dumb."

"ChatGPT can't learn, improve or even stay up to date without humans giving it new content and telling it how to interpret that content, not to mention programming the model and building, maintaining and powering its hardware," Nelson wrote in The Conversation.

"In my own testing, ChatGPT summarized the plot of J.R.R. Tolkien’s 'The Lord of the Rings,' a very famous novel, with only a few mistakes. But its summaries of Gilbert and Sullivan’s 'The Pirates of Penzance' and of Ursula K. Le Guin’s 'The Left Hand of Darkness' – both slightly more niche but far from obscure – come close to playing Mad Libs with the character and place names," Nelson explained.

"It doesn’t matter how good these works’ respective Wikipedia pages are. The model needs feedback, not just content."

Large language models don't actually understand or evaluate information, so they depend on humans to do it for them. "They are parasitic on human knowledge and labor. When new sources are added into their training data sets, they need new training on whether and how to build sentences based on those sources," Nelson added.

"They can’t evaluate whether news reports are accurate or not. They can’t assess arguments or weigh trade-offs. They can’t even read an encyclopedia page and only make statements consistent with it, or accurately summarize the plot of a movie. They rely on human beings to do all these things for them."

Then, Nelson highlighted, they paraphrase and remix what humans have said, and rely on yet more human beings to tell them whether they’ve paraphrased and remixed well.

"If the common wisdom on some topic changes – for example, whether salt is bad for your heart or whether early breast cancer screenings are useful – they will need to be extensively retrained to incorporate the new consensus."

In fact, recent investigation published by journalists in Time magazine found that hundreds of Kenyan workers spent thousands of hours reading and labeling racist, sexist and disturbing writing, including graphic descriptions of sexual violence, from the darkest depths of the internet to teach ChatGPT not to copy such content.

They were paid no more than US$2 an hour, and many understandably reported experiencing psychological distress because of this work.

"Large language models illustrate the total dependence of many AI systems, not only on their designers and maintainers but on their users," Nelson added. "So if ChatGPT gives you a good or useful answer about something, remember to thank the thousands or millions of hidden people who wrote the words it crunched and who taught it what were good and bad answers."

As the researcher said, far from being an autonomous superintelligence, ChatGPT is, like all technologies, nothing without us.