While the conversations they produce may seem astounding, ChatGPT, Bard and other contemporary "AI" chatbots are not magic. Rather, they are merely examples of what computer scientists call "large language models," or LLMs. Essentially, LLMs are trained to identify likely sequences of words, then generate coherent and convincing text.
In practice, ChatGPT and its ilk are really, really good at carrying on conversations and giving coherent, generally correct, responses to a wide variety of questions. Their uncanny text-generating prowess gives a beguiling illusion of intelligence that is convincing a growing number of people that LLMs now possess or soon will possess artificial general intelligence — the ability to do any intellectual task that humans can do, and the holy grail of artificial intelligence research. Those convinced include Ian Hoagarth, who penned a recent op-ed for the Financial Times titled "We must slow down the race to God-like AI."
ChatGPT's sentences are sometimes irrelevant, sometimes confusing, and sometimes incorrect — but always based on nothing more than statistically-identified word patterns.
Such concerns are almost certainly premature. That's because LLMs are still inexorably hobbled by the fact that they do not even attempt to understand what words or other data mean — which is why they cannot be trusted to make predictions or decisions with important consequences. In other words, they are very complicated remixers and refashioners of existing text and data, but there is nothing underneath the hood.
In order to constitute real artificial intelligence, an LLM would have to pass Winograd schema challenges — an updated version of the Turing Test, the challenge famously proposed by mathematician Alan Turing as a means of determining whether a computer was truly sentient. The Winograd schema challenge is a twist on that, designed for testing whether an artificial intelligence can understand language — specifically, the identification of the antecedent of an ambiguous pronoun.
Indeed, LLMs still struggle with Winograd schema challenges — even though dozens, if not hundreds, of such challenges are in the text databases LLMs train on. For example, here is one that I tested:
Gary: What does it refer to in this sentence? The trophy doesn't fit into the brown suitcase because it is too small.
ChatGPT: In the given sentence, "it" refers to the trophy.
Google's Bard also flopped:
Bard: In the sentence "The trophy doesn't fit into the brown suitcase because it is too small," the word "it" refers to the trophy. The sentence is saying that the trophy is too big to fit into the brown suitcase.
This sentence is an example of a Winograd Schema. A Winograd Schema is a type of natural language processing task that requires commonsense reasoning to answer. In this case, the commonsense reasoning required is that trophies are typically larger than suitcases, so if a trophy doesn't fit into a suitcase, it is because the trophy is too big, not because the suitcase is too small.
While it is interesting that the chatbot recognized that I was testing it with a Winograd Schema, it failed to actually exhibit "commonsense reasoning," as it claimed, and its conclusion is nonsense.
ChatGPT uses its vast database to identify related text that it can string together in prose that is grammatically correct, turgid, exceptionally verbose, and devoid of any understanding of the question or its answer.
Humans can easily answer such questions because, unlike computers, they know how words relate to the real world. To paraphrase Oren Etzioni, how can computers take over the world when they can't even figure out what "it" refers to in a sentence?
I want to focus here on a more complex example that reveals a great deal about the abilities and limitations of LLMs. Here's another interaction I had with these LLMs:
Gary: Suppose there is a wooden box with three cards inside. One card is red on both sides; one card is blue on both sides; and the third card is red on one side and green on the other side. If you pick a card randomly and look at a side randomly and see that it is red, what is the probability that the other side of the card is green?
This is a variation of the well-known Bertrand's box paradox and several related probability puzzles that are surely in the databases pillaged by ChatGPT, Bard, and otherLLMs.
ChatGPT gave a very long-winded answer, as it is prone to do, which I have broken into nine parts (Bard gave a shorter, but also incorrect, answer):
1. Let's first consider the three possible cards:
- Red on both sides
- Blue on both sides
- Red on one side and green on the other side
We know that one side of the card we picked is red. This eliminates the possibility of card 2, as both sides of that card are blue. Therefore, the card we picked is either card 1 or card 3.
The first bullet point of ChatGPT's answer is factually correct but quite pedantic, and doesn't really help us get to the correct answer. It is this kind of reasoning that leads many people to believe that the answer is 1/2 because there are two possibilities for the card we selected — but ChatGPT doesn't draw that conclusion.
ChatGPT continued:
2. Now, let's consider the probability that the other side of the card is green, given that we know one side is red. If we picked card 1, then the other side must also be red, so the probability of the other side being green is 0. If we picked card 3, then the other side must be green, so the probability of the other side being green is 1.
3. Since we don't know whether we picked card 1 or card 3, we need to take into account the probability of picking each card. Since we picked a card randomly, each card has a probability of 1/3 of being picked. Therefore, the overall probability of picking card 1 and seeing a red side is (1/3)(1/2) = 1/6, while the probability of picking card 3 and seeing a red side is (1/3)(1) = 1/3.
Bullet point 2 is correct but, if spoken by a human, would seem like stalling.
Bullet point 3 starts in the right direction, but then mixes up cards 1 and 3.
4. Using Bayes' theorem, the probability that the other side of the card is green, given that we know one side is red, is:
P(green side | red side) = P(red side | green side) * P(green side) / P(red side)
5. We already know that P(red side) is the sum of the probabilities of seeing a red side for cards 1 and 3, which is 1/6 + 1/3 = 1/2. We also know that P(green side) is the probability of picking card 3, which is 1/3.
6. The only missing piece is P(red side | green side), the probability of seeing a red side given that the other side is green. Since we know that the other side of card 3 is green, this probability is 0.
Part 4 recognizes that Bayes' theorem can be used to solve the problem, no doubt based on the fact that, in its training data, Bayes' theorem is often used to solve these kinds of problems. However, the definitions of "green side" and "red side" are confusing. Better labels would be "back of a selected card is green" and "front of a selected card is red."
In part 5, ChatGPT incorrectly states that "P(green side) is the probability of picking card 3, which is 1/3." The correct probability that the back of a selected card is green is (1/3)(1/2) = 1/6.
Bullet point 6 is nonsense. The probability that the front of a selected card is red given that the back is green is 1, not 0.
7. Plugging these values into Bayes' theorem, we get:
P(green side | red side) = 0 * (1/3) / (1/2) = 0
8. Therefore, the probability that the other side of the card is green, given that we know one side is red, is 0.
9. In other words, seeing a red side does not provide any additional information about whether the other side is green, because the card could be either card 1 or card 3 with equal probability.
Part 7 plugs the (incorrectly derived) probabilities into Bayes' theorem. The correct answer is (1/6)(1)/(1/2) = 1/3.
Bullet point 8 asserts a conclusion that any semi-alert person would recognize as obviously wrong. The probability that the back of the selected card is green is surely not zero.
And part 9 adds gratuitous, contradictory explanations that contradict the zero-probability answer it just gave. Yet ChatGPT does not recognize that these are contradictions. If seeing a red side provides no additional information, then the probability that the red/green card was selected is 1/3. If the selected card can be either card 1 or card 3 with equal probability, then the probability that the red/green card was selected is 1/2.
How might an intelligent human answer the question? Very simply: using logical reasoning. There are three ways to choose a red side and, in one of these three cases, the other side is green. So, the probability is 1 in 3. An attractive, but incorrect, answer is that seeing the red side rules out Card 2 so the probability is 1 in 2.
What does ChatGPT do instead? It uses its vast database to identify related text that it can string together in prose that is grammatically correct, turgid, exceptionally verbose, and devoid of any understanding of the question or its answer. The sentences are sometimes irrelevant, sometimes confusing, and sometimes incorrect — but always based on nothing more than statistically-identified word patterns. ChatGPT's zero-probability answer is clearly wrong, but it has no way of recognizing its error because it literally does not "know" what it is talking about.
There are consequences to the misconception that these chatbots are anywhere close to sentient: unjustly inflated stock prices, a general mania in the tech industry to advertise one's software as employing "AI," and a cultural obsession over the potential of what, as I've explained, is a glorified textual re-sequencer. But perhaps most importantly, the hype over LLMs threatens to create a new tech bubble — which, when burst, will have the same sorrowful consequences as any other financial bubble.
