Excitement, Glee, Fear, Envy, Envious dismissal, Caution & Warning – this is how the world has responded to chatGPT, the ‘latest’ AI platform OpenAI, which has received funding from Microsoft, among others, since it became downloadable in September 2022. Not surprisingly, there have been (and will continue to be) varied responses depending on the individual and also because AI is not just technology but a business in technology. In what follows, I have woven together multiple perspectives, gathered through asking different questions, to understand the foundations of chatGPT and what it means and forebodes.
Amidst all the euphoria, it was almost amusing to read Yann LeCun, the chief scientist at Meta (formerly Facebook) dismissing chatGPT as ‘not particularly innovative’ (https://www.techcircle.in/2023/01/25/chatgpt-is-not-particularly-innovative-meta-chief-ai-scientist), quoting a report by ZDNet, where he said that “In terms of underlying techniques, ChatGPT is not particularly innovative”. According to Techcircle, “he later tweeted that ChatGPT and other large languages are the results of decades of contributions from various people and no one AI lab is significantly ahead of others” (URL as above). Incidentally, Professor Sutton (www.incompleteideas.net) prophesied more than two decades ago that the future of AI was in NLP.
What is new
Just four years ago in 2018, researchers at Google Language AI launched BERT (Bidirectional Encoder Representations from Transformers) in a paper. It too created a lot of excitement but within the NLP community as it adopted what was then an innovative approach to language modelling, and developments since seem to suggest that it is used by firms working in this field, but along with other platforms.
From 2018 to 2022, what has changed is the mass media coverage of chatGPT, vaulting beyond the closed walls of NLP professionals and reaching out to lay people, who, hopefully, will drive the market. Although chatGPT is free now, its promoter, OpenAI, has already announced it plans to start charging. Microsoft alone has invested $1 billion and has announced another $10 billion! Even as Google and others working on similar efforts must be studying what this means for them, one media report says that “Gmail developer Paul Buchheit has predicted that Google may have only a year or two before total disruption following the launch of the AI chatbot ChatGPT”. According to him, the AI would do what Google did to Yellow Pages. He makes a specific observation: “AI will eliminate the Search Engine Result Page, which is where they make most of their money. Even if they catch up on AI, they can’t fully deploy it without destroying the most valuable part of their business”. (https://www.datanami.com/2023/01/06/the-drawbacks-of-chatgpt-for-production-conversational-ai-systems/).
People who follow Big Tech will know that, besides the original search engine, Google has not itself developed anything new but has acquired everything with its formidable financial power – Gmail, YouTube, Android, autonomous vehicle systems and so on. You can certainly expect Google to put all its might into developing a competing alternative to chatGPT which will nullify, if not eliminate, any threat to its search engine. Sundar Pichai, Google’s CEO, has already announced that it will push for a competing AI chatbot (https://www.thehindubusinessline.com/info-tech/google-plans-to-demo-ai-chatbot-a-chatgpt-competitor-report/article66419485.ece).
What it is and not
GPT is Generative Pretrained Transformer and has been available as GPT 3.5 (which came earlier) and chatGPT, modelled on GPT 3.5. Sam Altman, CEO of OpenAI which has developed chatGPT, has warned through a Tweet (on December 11, quoted in the article mentioned above – https://www.techcircle.in/2023/01/25/chatgpt-is-not-particularly-innovative-meta-chief-ai-scientist) that ChatGPT can have flaws: “ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness”, and that a lot of work needs to be done on the robustness and truthfulness aspects of the tool. Another article corroborates what he says by pointing out that GPT 3.5 “and its neural network has more layers than ChatGPT. GPT-3.5 was developed as a general language model that can do multiple things, including translate language, summarize text, and answer questions. OpenAI has provided an API interface for GPT-3.5, which provides a more efficient way for developers to access its capabilities. ChatGPT is based on GPT-3.5, but was developed specifically to be a chatbot. A limiting factor is that ChatGPT only sports a text interface; there is no API”. Incidentally, OpenAI has confirmed launch of chatGPT API, a business integrated platform which will be integrated to existing apps and services (https://www.livemint.com/technology/tech-news/chatgpt-plus-subscription-introduced-all-you-need-to-know-on-price-and-features-11675317411541.html).
Datanami’s article says that “ChatGPT was trained on a large set of conversational text and is better at holding up a conversation than GPT-3.5 and other generative models. It generates its responses more quickly than GPT-3.5, and its responses are perceived to be more accurate. However, both models have a tendency to make stuff up, or “hallucinating” things, as those in the industry call it. Various hallucination rates have been cited for ChatGPT between 15% and 21%. GPT-3.5’s hallucination rate, meanwhile, has been pegged from the low 20s to a high of 41%, so ChatGPT has shown improvement in that regard” (https://www.datanami.com/2023/01/06/the-drawbacks-of-chatgpt-for-production-conversational-ai-systems/). For example, Analytics Vidya showed what absurdity can be generated from chatGPT:
(https://analyticsindiamag.com/a-closer-look-at-chatgpts-limitations-what-you-need-to-know/).
The role of human interaction
The role of human interaction in chatGPT generating texts is fundamental. Although it has been mentioned that human inputs have to be absolutely clear, structured to generate quality texts, this critical aspect seems to be lost in all the noise.
Sandeep Kaul, promoter of the start-up Hipla (https://hipla.io/) told me that chatGPT can be very good for certain text tasks as long as your inputs are clear and well-structured, which he found by using chatGPT to generate multiple marketing blogs for his start-up. An article in South China Morning Post quoted experts in the Asia-Pacific region as saying chatGPT is pedestrian and good for straightforward writings. Toby Walsh, a professor of Artificial Intelligence at the University of New South Wales (UNSW) in Sydney, commented that “ChatGPT writes very pedestrian, and often balanced essays, as it has been carefully designed to offer middle-of-the-road responses. Actually taking a position, or debating a particular side of a topic, will be quite difficult for ChatGPT”.
Analytics Viday found that chatGPT was not good at summarizing – “Oftentimes, the software fails to understand the context of the paper and spews out unrelated and incoherent results”. It tested the AI for other dimensions: “Something that OpenAI does not reveal but gets automatically exposed is ChatGPT’s mathematical and analytical capabilities” doing fine with simple addition and subtraction but “as soon as you add multiple layers to the calculation or a predictive problem, the chatbot boggles” (https://analyticsindiamag.com/a-closer-look-at-chatgpts-limitations-what-you-need-to-know/).
The problem could well lie with users or rather users’ understanding and expectations. The excitement generated largely by the way the mass media has covered chatGPT has led many to believe that chatGPT is the ultimate wonder AI which can do anything in text. But Jiang Chen, founder and vice president of machine learning at Moveworks, recommends against placing faith in just one system. He says: “Our strategy is using a different set of models in different places. You can use large models to teach your smaller models, and then the smaller models are much faster. For example, if you wanted to do a segmented search, you want to use…some kind of BERT model, and then run that as some kind of vector search engine. ChatGPT is too big for that.” According to him, “while the OpenAI models are much bigger and are trained on a much larger corpus of words, there’s no way to know if they’re the right words for a specific customer” (https://www.datanami.com/2023/01/06/the-drawbacks-of-chatgpt-for-production-conversational-ai-systems/). What he says a little later is intriguing: “The [ChatGPT] model is pretrained to encode all the knowledge that is fed into it. It was not designed to do any specific task itself. The reason it was able to speed up and achieve fast growth is because the architecture itself is actually simple. It’s layers and layers of the same stuff, so it’s kind of fused together. Because of that architecture, you know it learns something, but you don’t know where it encodes what information where. You don’t know what layers of neurons encode that specific information you want to inference it to, so it becomes more of a black box.” (URL as above)
Training and more training
As is true for any algorithm, chatGPT is trained on data, specifically the 2021 data on the internet. In technical terms, an algorithm is said to suffer from ‘overfitting’ when it functions well on training data but falls flat when used on data beyond that. This is a basic test for all algorithms. Normally, it takes an enormous amount of data to train algorithms and often many algorithms fail because they are unable to muster sufficient volumes of data for training. The advantage for language-based AI is that the whole internet can be data. Let me emphasize that chatGPT depends on what human beings have said and written; it is not creating a new text out of nowhere!
Everyone in this highly competitive field will watch how this AI platform will respond when it is ‘given’ more data but the history of algorithms does not offer a conclusive clue. Meanwhile, competing AI systems will become available, just as it happened in the case of search, where Google was not the first but became the most successful by a considerable distance, displacing the earlier search engines.
Just as I finished writing, I read that chatGPT will charge $20 a month but restricted to the US but will not stop the free use (https://www.livemint.com/technology/tech-news/chatgpt-plus-subscription-introduced-all-you-need-to-know-on-price-and-features-11675317411541.html). That was fast – from free launch to pricing in about four months. That shows great planning. Great battles lie ahead.
Takeaways
Probably the first AI engine offered to masses
Resounding initial success has taken even informed people by surprise
Sober response is likely only from the well-informed
Competing AI platforms will emerge
History of search engines shows that first mover advantage is no guarantee of success
