Demystifying ChatGPT: understanding how it works in plain English

One of the games you've probably played when you were young is the Word Chain game. It's a simple but fun game where players have to create a chain of words linked by some rules. There are different variations of this game, but in the version, I'm talking about, the first player will start with a simple phrase like "Yesterday, I went to the", and the second play must repeat that phrase and continue it by adding a word. So in this context, the next word might be "beach", "shopping mall", or some other place. The game ends when a player cannot think of a word that fits the rule.

Now let's imagine for a second that the second player is an AI. So I give it a phrase, and he writes the next word. But rather than taking turns, the AI keeps taking all the turns until it cannot think of a word that fits the rule. So if I type "Yesterday, I went to the", the AI will reply: "Yesterday, I went to the beach, had an ice cream but felt chilly, so I went home."

Well done, you've just played word chains with an AI!

Doesn't this game remind you of something? It probably does because this is precisely how chatGPT and other similar AI systems work. The approach is straightforward, and you've understood it in a few lines. That is why many top AI scientists are cautious about getting too excited with such techniques.

But I'm sure you're questioning whether there is more to it because it can't be so simple. Of course, there are a few other tricks that these algorithms use.

First of all, according to the Economist,  an adult English speaker has an active vocabulary of around 20,000 words and a passive one of approximately 40,000 words. In a lifetime, it is estimated that an average person speaks 860 million words. The AI we're talking about stores around 175 billion concepts, and it is estimated that the next-generation ones will reach the 100 trillion mark. So the difference between these AI and us is staggering in terms of coverage.

Second, whilst I can barely remember what I had for breakfast this morning, the AI can effectively remember everything up to extremely minute detail in a fraction of the time. Hence why it can recall various facts with extreme precision and detail.

Third, apart from complex statistical computations, these systems use what is known as an attention mechanism to try to guess the most plausible word. Essentially this means that through complex AI, they can determine whether a term fits a particular context or not. So in the sentence "The Nile is long, and many people take a stroll along the ", the AI knows from its vast knowledge that we're talking about a river, so it will suggest concluding the sentence by using the word "bank". What's interesting here is that apart from calculating the statistical probabilities of the next word, it also performs semantical disambiguation. Thus, it knows that in this context, the bank refers to the river bank, not the financial institution.

Fourth, an impressive feat of such an algorithm is that it produces original content. In the beginning, everybody expected to see highly-plagiarised texts, but this did not happen. The reason for this is that such algorithms have a property called temperature. Essentially, temperature tunes the level of creativity. So when the algorithm is trying to choose the next word, should it go for the highest-ranking one or maybe opt for something slightly less common? If we continue the previous example, rather than "bank", it might suggest "shore" or "edge". By doing so, it is constantly creating new sentences, even though we're asking it the same question!

Fifth, there's a final trick up their sleeves. The technology used is not new and was invented in 2017. There were various algorithms created, all of them impressive, but none of them was as good as chatGPT. The difference is not in the program but in the type of learning. Initially, the training was only performed on freely available text. So even though it was good, it was highly mechanical and felt unnatural. So chatGPT created a Word Chains game, where the algorithm produced various sentences for the initial phase, and humans ranked them. Keeping to the previous examples, it would create "The Nile is long, and many people take a stroll along the bank" and "The Nile is long, and many people take a stroll along the shore". A human being would then mark the first sentence as the preferred option. This information was then fed into chatGPT and used to adjust its preferences. Thus, it not only learnt to churn good results but chose the ones preferred by humans.

Finally, a limitation of chatGPT is that its database is restricted to information gathered until September 2021. So if anyone asks about the 2022 Qatari world cup, the program will have no knowledge. The new integration with the Bing search engine goes beyond this limitation. Microsoft tackled it smartly by submitting chats to the bing search engine first, retrieving the results and then using the GPT engine to combine everything. This means that technically, Bing can today answer precise questions about anything.

So as you can see, there's nothing magical or mystical about chatGPT and similar AI. On the contrary, it's all about using existing algorithms smartly. But as Arthur C. Clark, the famous Science Fiction writer, once said, "Any sufficiently advanced technology is indistinguishable from magic". Hence, when we start reading about; AI becoming sentient, turning evil, diverging into split personalities, or even falling in love with the user, we should think that this is utter nonsense. chatGPT is as conscious as my calculator. Of course, this doesn't mean that it won't happen in the future, but at the moment, we're still far, far away.