What Happens When Most Content Online Becomes AI-Generated?
Introduction
The recent breakthroughs in Generative AI have introduced publicly available AI models capable of producing highly realistic and complex text, image, and sound that revolutionize content creation.
These models were trained on large datasets scraped from the Internet. In the case of text data for instance, Large Language Models (LLMs) like ChatGPT were mostly trained on human-generated text found online.
Generative models have gained wide notoriety and fast adoption in society, to the extent that more and more AI-generated content ends up on the Internet, which is the major source of their training data.
We see a loop forming here: generative models will inevitably be trained on synthetic data produced by Generative AI and not by humans. Which brings us to the question: if that happens, how will the models behave?
Generative models and degeneration
What will happen to GPT-{n} once LLMs contribute much of the language found online?
That is the question asked by a team of researchers that published the paper The Curse of Recursion: Training on Generated Data Makes Models Forget (I. Shumailov et al., 2023).
The authors experiment with making models learn from the data they generate over several iterations. They apply this on Gaussian Mixture Models (GMMs), Variational Autoencoders (VAE) and Large Language models (LLMs). For the three types of models, this leads to a phenomenon they call "model collapse":
a degenerative process whereby, over time, models forget the true underlying data distribution, even in the absence of a shift in the distribution over time.
Model collapse characterizes the behavior of models forgetting the tails of real distributions (i.e. the improbable, less frequent and yet important events) and over-representing the center of the distribution. As the process repeats, models converge to a distribution that has little similarity with the original one.
In the paper Self-Consuming Generative Models Go MAD (S. Alemohammad et al., 2023), other researchers similarly describes this gradual divergence away from original distributions as "Model Autophagy Disorder" (MAD):
[…] without enough fresh real data in each generation, future generative models are doomed to Model Autophagy Disorder (MAD), meaning that either their quality (measured in terms of precision) or their diversity (measured in terms of recall) will progressively degrade.
The conclusion is clear: generative models tend to deteriorate when they are primarily trained on the content they generate. Given that the majority of their training data comes from the Internet, how much of a risk does this degenerative process represent?
When human-generated content grows scarce
If the proportion of human- vs. AI-generated content remains as it is today, with a clear majority of online content created by humans, the situation should not be worrying and the performance of Generative models should not face the degeneration described earlier.
However, there is a legitimate concern over the prospect of an Internet dominated by AI-generated content. If we focus on text generation, the main argument supporting this projection is the convenience offered by LLMs like GPT. These tools address real pain points when it comes to writing: finding the right words, improving style, getting inspiration. Those who have tried them can attest how tempting and addictive it is to rely on these tools: why struggle when we can effortlessly prompt an LLM to generate content that is better both in style and time-efficiency?
So if human-generated content becomes less prevalent, what measures can be taken to mitigate the risk of exposing Generative models to AI-generated content? One possible approach is to attempt to counter the trend by encouraging content creation without AI tools. Another approach involves developing methods to detect AI-generated data and filtering it out during the model training process.
The idea of the first approach is to find ways to incentivize creators to use generative models less frequently or in fewer contexts. However, the feasibility of this approach may be limited. Even if we could find such incentives (e.g. a reward or a sanction), the challenge lies in verifying whether a piece of content is genuinely human-generated or not. This leads us to the second approach, which also relies on the ability to distinguish human-generated from AI-generated data.
How to detect AI-generated data
In 2023, the company AI21 Labs led the largest Turing test to date: over 1.5 million users engaged in an online chat with either a human or an AI chatbot and were asked to guess who they spoke to. The task proves to be non-trivial as 68% of people guessed correctly, showing how well generative AI can mimic humans.
With that in mind, let's review some available techniques to identify AI-generated data.
Watermarking
The main approach discussed in the literature to distinguish human-generated from AI-generated data is watermarking. Watermarking is the process of adding hidden signals in the data that are invisible to humans but detectable by algorithms.
Watermarking would enable to easily detect AI-generated data, but it only works if Generative AI providers add it to their products. This is not the case today, and it might be too optimistic to expect that any time soon.
In his open letter Watermarking is a No-Go, Andrew Ng is not optimistic about the adoption of watermarking. He explains how, in the current GenAI race, watermarking appears like a competitive disadvantage for AI companies.
Using ML classifiers
An alternative to watermarking would be to train a Machine Learning classifier that learns to label whether a content is AI-generated or human-generated.
Systems performing this task already exist, such as GPTzero. However they show a high error rate and fall short of offering a robust solution. Detecting AI-generated data proves to be a difficult task, even for machine learning models.
A zero-shot approach
Another approach discussed in the paper DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (E. Mitchell et al., 2023) does not require any training and is based on observing the curvature of the log probability function of LLMs.
The criterion discovered by the authors is that AI-generated text tends to occupy regions with negative log probability curvature, as opposed to human-written text.
This seems like a promising technique, although we may have doubts concerning its scalability: it is not model-agnostic (it detects whether a content was generated by a specific LLM, not by any given LLM) and requires to have access to the model's log probabilities.
Conclusion
We've learned about the importance of training generative models on human-generated data to avoid the decline of their performance. This poses a significant challenge, particularly as the Internet is their primary source of training data, and as more an more content online is AI-generated.
There are two approaches to address this issue: reversing the trend by promoting human content creation, and preventing models from being trained on synthetic data. However both solutions raise complex, open-ended questions. How can we incentivize online content creators to rely less on AI? How can we establish reliable, scalable methods for detecting AI-generated content?
On a promising note, the White House made a noteworthy announcement in July 2023. Seven major AI companies – Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI – agreed to respect a list of responsible-AI commitments which include watermarking. The future will reveal whether these commitments are upheld.
To go further
- Publication of AI21 Labs where they analyze the results of their "Human or Not?" experiment and explain their methodology
- Sparks of Artificial General Intelligence: Early experiments with GPT-4, a thorough investigation on GPT-4 and an interesting discussion about AGI, by a team of researchers at Microsoft
- Watermarking is a No-Go, a letter by Andrew Ng about how financial "incentives in the competitive market for generative AI make [watermarking] adoption challenging"
