RLHF – reinforcement learning through human feedback. This fanciful and seemingly innocuous phrase used in GenAI to moderate the output it generates hides a dark truth. GenAI needs content-moderation on a massive scale, because its inputs are the ‘vast corpus of humanity’, making it impossible to control. Some writers glibly talk about how such moderation, practised through the use of third parties, bring about ‘ethically aligned’ output, completely ignorant, wilfully or otherwise, of a vast environment of clearly unethical practices. We owe this revelation to an old institution: good journalism.
There is a difference in the data used to train GenAI models and Agentic AI models and that leads to a decision on what to control: input or output. Ryan Kolln, CEO, Appen, a platform for connecting Silicon valley companies with data workers, describes it thus: “In the traditional AI sense, you control the output by controlling the inputs, because it only learns from the examples you are giving it. The challenge with GenAI is the inputs are the vast corpus of humanity. So, you need to control the output”. (Quoted in Karen Hao’s book – Empire of AI, page 137)
The ‘vast corpus of humanity’ is the key phrase because it would envelop everything that is brought under the rubric ‘user-generated content’, some of which are explicitly sexual and violent. In 2024, a Stanford University study analysing LAION-5B, a five billion images dataset used to train Stable Diffusion, found that it contained thousands of images of verified and suspected child abuse images. (Quoted as above) {Stable Diffusion is a technique to generate images from text inputs, but is a slow process and needs a lot of memory and hence began to use a technical process called Latent Diffusion which cuts down the memory requirements. You can find more information here – https://medium.com/@onkarmishra/stable-diffusion-explained-1f101284484d, and https://sertiscorp.medium.com/latent-diffusion-models-a-review-part-i-d0feacc4906.
Controlling input & output
To repeat what we noted above, for GenAI, the inputs are the vast corpus of humanity, which creates data, both text and visual, from the trivial to the absurd to the false to the objectionable and to the outlandish conspiracies. GenAI feeds off on this vast reservoir making it deeply and fundamentally parasitic; when faced with topics which are under-represented, it can err widely. As Karen Hao remarks in her detailed and insightful book, “The AI industry calls these inaccuracies ‘hallucinations’ (page 113). An article in Futursense.com says, hallucination “simply refers to the tendency of AI to generate incorrect or fabricated information that may sound plausible”.
Anyone who has used predictive text in smartphones would have often experienced widely off the mark suggestions, because it is based on probability – predicting the next word or the next sentence, based on the law of large numbers. However, since everyone does not think the same way, text generators can err widely.
An outstanding article titled ‘Towards a theory of AI errors’ by Veronica Barassi points out that one of the problems in GenAI models is not just that they are trained on billions of paarmeters but that they rely on transfer learning – taking what they learned from one task (say object recognition in images) and apply it to another task (say, activity recognition in videos). (https://hdsr.mitpress.mit.edu/pub/1yo82mqa/release/2)
Output moderation and exploitation of labour
It became clear to GenAI companies that controlling input was an impossible task, thus shifting it to control (or rather audit) output in GenAI. And, this has opened up a new business in data (output moderation) built on exploitation of labour in poor countries, creating one of the tragic ironies of the AI world – a large number of people needed to go through the datasets and use their feedback to improve the output of an AI! The AI industry christened it with an academic-sounding expression – RLHF, reinforcement learning from human feedback)!
Billy Perrigo, a journalist with Time magazine, who unearthed this unethical practice, says that “ Content moderators are the front-line workers of the internet: the people who remove traumatic content from social media platforms and AI datasets”. Karen Hao points out that “OpenAI would hire workers in Kenya for an average of less than two dollars an hour to build an automated content-moderation filter, a revelation first reported by Time magazine correspondent Billy Perrigo”, and adds that it would employ thousands of contractors globally as intermediaries to employ workers. [Empire of AI, page 137] Incidentally, I wonder what Professor Rich Sutton, father of reinforcement learning would say. [In earlier articles, I have referred to this distinguished Professor and his website – http://incompleteideas.net/].
This practice of content moderation has disastrous mental health consequences for those who have to go through vast amounts of disturbing content, a fact that is completely absent in all the tech media coverage of AI and other writings on RLHF. Billy Perrigo adds that “Now, new research suggests that African moderators have it worse than their colleagues in Asia, Europe, and the Americas when it comes to their mental health. A survey of 134 moderators led by researchers at the University of Minnesota finds that 52% of surveyed African content moderators met thresholds for probable clinical depression, and 55% had significant levels of psychological distress. Some 28% reported using drugs or medication to cope with their symptoms”.
OpenAI, Meta, TikTok all have resorted to this practice as Perrigo records shocking details in his research: “The researchers carried out supplementary interviews with 15 moderators to answer the question of why African content moderators’ well-being scores were so low. They found a host of working conditions that will be no surprise to people familiar with the topic. These include low pay, deceptive recruitment practices, stigma, non-disclosure agreements, precarious employment, inadequate wellness programs, and companies’ frequent failure to renew expired work permits that can trap workers in a foreign country away from their families”. (URL as above)
This practice of content-moderation has been going on for at least five years. In 2022, Perrigo unveiled the story of an African student who undertook this, along with many others, for Facebook. Tech companies use an intermediary so as to claim ignorance of any unethical practice by a ‘third party’, some of whom are ‘respectable’ NGOs.
And yet, we have a post in LinkedIn by Anand Ramachandran on RLHF which glibly observes that “This methodology is transformative because it allows for models to better align their outputs with human expectations, preferences, and values, resulting in outputs that are more contextually appropriate and ethically aligned”. This kind of ill-informed and Ostrich-like response to RLHF is not an exception but completely normal! Do we really live in an age of information?
The clever use of third parties including seemingly respectable NGOs puts a veil over the exploitation of labour, which is common in the field of environment, specifically mining. The mainstream media steadfastly remains blissfully ignorant, and yet we are made to drown in ‘profound debates’ over ethics in AI! This reminds me of the Theatre of the Absurd!
