Should we be surprised at the power of GenAI to deceive? As a growing body of serious research is trying to find convincing answers, it also show how important the role of mainstream and specialized media is, as vehicles of communication.

Editing is central to how narratives are created, be it films, journalism, history, legal process, and even science, with deception an inevitable part of such ‘pursuits’, creating ‘any reality’ through appropriate placement, addition or deletion of words, images and data, separately and together, with intent to persuade or mislead. DeepFake is the latest editing technique.

Power to deceive

Deception is a derived function of language as Nathan Oesch of the Department of Experimental Psychology, University of Oxford, has argued (https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.01485/full). This power to deceive has been co-present with the birth and spread of the printing press culminating in mass circulated newspapers and books. It has become easier and faster with the advent and growth of electronic media and more so now with software and algorithms, aided and abetted by people’s readiness to believe, despite all the cautionary warnings.

In ‘Is AI lying to me? Scientists warn of growing capacity for deception’ (https://www.theguardian.com/technology/article/2024/may/10/is-ai-lying-to-me-scientists-warn-of-growing-capacity-for-deception), The Guardian reports on an analysis “by Massachusetts Institute of Technology (MIT) researchers, (which) identifies wide-ranging instances of AI systems double-crossing opponents, bluffing and pretending to be human. One system even altered its behaviour during mock safety tests, raising the prospect of auditors being lured into a false sense of security”.

Dr Peter Park, an AI existential safety researcher at MIT and author of the research, warns that it will only get worse. He was prompted to “to investigate after Meta, which owns Facebook, developed a program called Cicero that performed in the top 10% of human players at the world conquest strategy game Diplomacy. Meta stated that Cicero had been trained to be “largely honest and helpful” and to “never intentionally backstab” its human allies”. There were comparable issues with other systems. ‘Broken Code’ by Jeff Horwitz offers a narrative of Meta’s chequered history in this pursuit.

The use of the word deception is problematic for two reasons. First, it suggests wilfulness, that the AI agent ‘knowingly misled’ the user, which ascribes to the agent a level of intelligence that is not supported by any of the current understanding of the model. Second, the element of researchers’ surprise suggests that such an outcome was not expected.

Opaque

Professor Rayid Ghani, a Distinguished Career Professor at the Heinz College of Information Systems and Public Policy and in Carnegie Mellon University’s Machine Learning Department, has identified a crucial piece of the missing link. He explains that “There are two components to these systems. Component one is, it takes all the data that’s on the Internet and predicts the next word. And we kind of understand what that is. Component two is, it’s been trained from human feedback for the last few years, and nobody outside OpenAI knows how that is done and what its impact is.” (https://www.heinz.cmu.edu/media/2023/July/generative-ai-is-a-math-problem-left-unchecked-it-could-be-a-real-problem).

In a later essay (February 7, 2024), ‘Demystifying generative A: true, false, uncertain’, Laure Soulier, Senior Lecturer, Machine Learning and Information Access Team, Sorbonne University (https://www.polytechnique-insights.com/en/columns/science/demystifying-generative-ai-true-false-uncertain/), observes that “generative AIs are not always very stable. Try it out with ChatGPT: ask the same question but vary the wording, and you’ll sometimes get different answers! These systems are based on mathematical operations that transform information into high-dimensional vectors, which makes them difficult to explain. Research is currently underway on this subject”. Many studies have found that the phrasing of questions makes a decisive difference to the output generated by the GenAI agent.

Plausibility, not truth

Contrary to popular misconception and even MIT’s expectations, Soulier makes an arresting point: “generative AI does not aim to deliver the truth, but to maximise plausibility, based on its training data. It sometimes produces false correlations between words”. And often, it can lead to ‘hallucinations’ – incorrect responses or incoherent images.

IBM explains that “AI hallucination is a phenomenon wherein a large language model (LLM)—often a generative AI chatbot or computer vision tool—perceives patterns or objects that are nonexistent or imperceptible to human observers, creating outputs that are nonsensical or altogether inaccurate”. It offers some striking examples.

  • Google’s Bard chatbot incorrectly claiming that the James Webb Space Telescope had captured the world’s first images of a planet outside our solar system.
  • Microsoft’s chat AI, Sydney, admitting to falling in love with users and spying on Bing employees.
  • Meta pulling its Galactica LLM demo in 2022, after it provided users inaccurate information, sometimes rooted in prejudice.

Many of these issues have since been addressed and resolved, but the basic truth is that, “even in the best of circumstances, the use of AI tools can have unforeseen and undesirable consequences” (https://www.ibm.com/topics/ai-hallucinations).

Another article (https://www.telusinternational.com/insights/ai-data/article/generative-ai-hallucinations) revealed that “a New York attorney representing a client’s injury claim relied on a conversational chatbot to conduct his legal research. The federal judge overseeing the suit noted that six of the precedents quoted in his brief were bogus. It turns out that not only did the chatbot make them up, it even stipulated they were available in major legal databases”.

The future

Formal structures are the bedrock of any computer-based intelligence. Since every GenAI agent learns only through statistical patterns identified during training, such formal structures have to be continuously refined with every human feedback. As professor Ghani observes, how feedback affects the performance of the AI agent is an unknown. The question is whether such refinements are sufficient, because even though there is a structure to every language and its use, we cannot escape its inherent rich and layered ambiguity, which extends even to context, which, often, is a process of discovery and choice, not a given.  

Since Covid and now GenAI, competing research institutions and universities have taken to mainstream media to communicate their work, as they want to be seen as fountains of knowledge. It is not a coincidence that both are enormous businesses. The rise in the number of competing GenAI products will turn this competitive space into a veritable battlefield. Can the mainstream media shoulder the enormous burden this places on them?