The word hallucination is not very accurate for ChatGPT or other generative AI. A generative AI system produces output based on its training, so there is no "perception of something not present" (the definition of hallucination). A more accurate term would be fabrication. Generative AI systems generate novel output, and sometimes it’s pure fabrication.
Generative AI is extremely good at spelling words correctly using an internal dictionary. Even when a user inputs misspelled words, the generative AI system can figure out the right words using a dictionary and a little context. If a user misspells the word halucination, it is easy for AI to fill in the missing l. For a slightly more interesting example, consider the typed sequence of letters cloes. Transposing e and s would yield close, whereas inserting th would yield clothes. How does AI decide which is correct? A few words of context around a misspelled word are generally enough for AI to figure it out.
Grammar has a slightly higher level of complexity. Within a given language, there are rules about how the words may be combined to make meaningful sentences. Generative AI is very good at grammar because the rules are relatively simple (e.g., subject, then verb, then object). It’s also easy to train generative AI systems in many different languages. If a user switches languages mid-stream, the system can respond using the most recent language.
It’s clear that generative AI gets high marks for spelling and grammar. Conversing with generative AI is like talking with a smart, highly educated, and experienced adult.
So why do these systems tend to fabricate novel content in some situations, despite having complex models trained on huge data sets?
It’s helpful to look at a couple of examples. In several instances, attorneys have used ChatGPT to write briefs that were submitted to courts. On the surface, the briefs looked reasonable and cited relevant precedential opinions (e.g., in a case that involved an airline, the cited cases involved airlines). Similarly, in some academic research submissions, ChatGPT has generated citations from relevant published articles in well-known journals. In these examples, however, the citations to court decisions or academic articles were complete fabrications.
Based on the mountains of training data, generative AI systems have learned what citations are supposed to look like and have responded by generating text that looks like real citations. This is the essence of fabrication: the systems are functioning exactly as they were trained. Unlike human lawyers or academics, though, generative AI systems have not learned that a citation has to correspond to a specific real thing. In human development, children generally know the difference between the truth and a lie by age 4 and know that they are supposed to tell the truth. There is as yet no corresponding developmental stage or training period for AI to appreciate the distinction.
Current generative AI systems are essentially savants: brilliant at certain tasks but profoundly immature in deeper conceptual or ethical thinking. We have to be aware that the development of AI tools is not analogous to human development. Though talking to AI might look and feel like talking to a person, AI is not human and does not have a conscience—at least, not yet.