Imagine you’re scrolling through a video platform. Five videos, five different creators, five completely different styles. But two minutes into each one, they’re all pushing the exact same fake giveaway. That’s not a coincidence – that’s Generative AI at work. One attacker, one scam, thousands of unique-looking versions, invisible to moderation systems built to catch just one at a time.
So, What’s Actually Going On Here?
Every day, online platforms receive millions of new pieces of content. Among them are AI-generated videos that may look completely different but are actually created to spread the same scam.
Generative AI has completely changed the way content gets made. Today, anyone can generate videos, images, voice recordings, or articles within minutes. While this has opened up a lot of exciting possibilities, it’s also created a serious problem for online platforms.
Earlier, spammers had to copy and paste the same message again and again – and that repetition is exactly what made them easy to catch. Now, they don’t need to repeat anything. Attackers can use AI to generate thousands of different versions of the same scam: different faces, different voices, different wording – but the goal behind every single one stays exactly the same.
This is exactly why traditional content moderation is struggling against AI-generated spam. The systems built a few years ago were designed to catch repeated, identical content. But when every piece of spam looks slightly different, “repeated” stops being something you can detect just by comparing files.
A research team at Google recently published a paper on this exact problem: “Scalable Detection of Adversarial Synthetic Slop and Coordinated Media Abuse.” The paper proposes a detection system called S-CTS – the Scalable Cluster Termination System – built around a different way of thinking about spam detection.
In this article, we’ll break down why AI-generated spam has become so hard to catch with traditional content moderation, and how the researchers propose solving it.
First, Let’s Talk About “AI Slop”
Here’s a term you’ll keep running into throughout this article – so let’s get it out of the way early.
Large volumes of low-quality content created using AI, not to educate, entertain, or genuinely help anyone – but to farm views, spread misinformation, push scams, or manipulate people into clicking, sharing, or trusting something they shouldn’t.
Same Scam, Multiplied
AI-generated videos aren’t new. What’s changed is how easily one attacker can produce many of them, and how easily each one can look completely unrelated to the rest.
A few years ago, even with AI tools, putting together a convincing fake video took real effort – writing a script, generating visuals, checking whether it looked believable. Now, generative AI can produce a new “wrapper” around the same scam in minutes: a different face, a different voice, a different script structure, automatically.
- One video features a “finance expert” claiming a cash prize.
- Another shows a “customer support agent” asking you to verify your account.
- A third has a “brand representative” offering an exclusive reward.
- A fourth mimics a well-known public figure endorsing a crypto giveaway.
- A fifth pitches a fake celebrity-backed investment scheme.
Individually, each video looks harmless and unrelated. But they’re all coming from the same attacker, pushing the same scam – just wrapped differently, at a speed that simply wasn’t possible before.
The paper describes this behavior as “adversarial adaptation” – attackers continuously tweaking their output specifically to stay just under a platform’s violation threshold, so no single piece of content looks bad enough to get removed on its own.
Why Traditional Content Moderation Fails Against AI Spam
Most moderation systems work on one simple idea: look at a piece of content, check if it breaks the rules, then act on it. This idea works fine when spam looks the same every time – like the same message being copy-pasted again and again. But it completely breaks down once every piece of spam looks a little different. And according to the paper, this is the real, deeper problem with how content moderation has traditionally worked.
To understand this better, let’s look at the tools platforms have relied on so far – and why each one struggles against AI-generated spam.
Just like every person has a unique fingerprint, every file can be given a unique code. If a file gets copied exactly, its code stays exactly the same too – so it’s easy to spot. But if even one small thing in the file changes, the code changes completely, just like a fingerprint scan failing to match if it’s smudged even slightly.
- Cryptographic hashing checks a file’s digital fingerprint against known bad content – if the fingerprint matches, the system catches it instantly. The problem is, it only works on the exact same file. With AI-generated spam, every new version – a different face, a different voice, slightly different wording – becomes a brand-new file with a brand-new fingerprint. So even though the scam is the same, the system has nothing to match it against.
- Metadata-based detection looks at details around a file – not what’s inside the video, but things like when it was uploaded, what device or app it came from, and how often that account is posting. The problem is, it only checks this for one account at a time. So if one account uploads a few videos a day, using a normal-looking pattern, it doesn’t raise any flags. But if 200 different accounts are all doing this – and they’re secretly connected, run by the same attacker – this method has no way of spotting that connection, because it never compares accounts to each other. It only ever looks at one account in isolation.
- Individual content review – Whether it’s a human or a machine, this method checks one video or post at a time and asks a simple question: does this break the rules? It doesn’t check who posted it or whether something similar already exists. This worked fine when spammers just copy-pasted the same message everywhere – catching it once meant catching all the copies too. But it fails against AI-generated spam, because every version looks different. Each video passes the check on its own, since none of them look like a repeat – even though hundreds of them together are the same scam, just wrapped differently.
The Proposed Solution: S-CTS
Everything in this paper comes back to one core idea:
Old systems ask: “Is this one piece of content spam?”
S-CTS asks: “Are these accounts working together?”
Instead of checking one video at a time, S-CTS looks at the whole group of accounts behind it. It checks if these accounts are connected to each other and posting similar AI-made content. If they are, the system reviews and takes action on the entire group together – not one video at a time.
This “both must match” rule is important. It means one person using AI tools on their own won’t get caught by this system, even if their content looks a bit AI-made. S-CTS only targets groups of accounts working together to mass-produce content – not individual creators.
How S-CTS Actually Works
Component 1: Finding the Bot-Net
The first part, called Ψ_A in the paper, looks for accounts that seem to be run by the same person or script. It doesn’t look at the content at all – only at behavior. Things like how the account uses its API, how often it posts, and other patterns that show up when AI tools are used to publish content automatically. Accounts that show similar behavior get grouped together. The paper calls this group a “Generation Cluster.”
Component 2: Scoring the Content
The second part, called Ψ_C, looks at the actual content – the text, video, and other media. It checks the content against Google’s quality standards and looks for what the paper calls “Generative Artifacts” – small signs that show up when content is mass-produced by AI, even if each video looks different on the surface.
A group of accounts only gets flagged when both checks agree: the accounts are behaving like a connected network, and the content they’re posting shows signs of being mass-produced.
A Two-Stage Architecture for Processing Video at Scale
This part of the paper is worth understanding. Running a full AI model on every single video, frame by frame, at the scale of a big platform would cost too much. So the researchers built a two-step system instead.
Stage 1 – Turning Video Into Text
Instead of checking each frame for visual glitches, this step turns the video, audio, and text into a short summary – basically describing what’s happening in the video, in words. It looks for repeated, template-style storylines and checks how fast an account is posting. A creator needs time to write, record, and edit – an automated system just keeps publishing.
Stage 2 – Making the Call
A second, LLM-based classifier reads that short summary and decides what to do – remove it, approve it, or send it to a human for review. Since it’s working with a short summary instead of the full video, this step is much faster and cheaper to run at scale.
LoRA + APO: Staying Ahead of New AI Models
New AI models keep coming out all the time. When a new one launches – like Sora or Kling – scammers start using it right away. The problem is, each new AI model creates content in its own way, with its own patterns. The detection system was trained to recognize patterns from older models – so when a new model shows up, its content looks unfamiliar, and the system doesn’t know what it’s looking at yet.
Retraining the entire system from scratch every time this happens would take too much time and cost too much money. So the paper suggests using two methods together:
- LoRA (Low-Rank Adaptation): A big AI model has millions of settings inside it that make it work. Instead of changing all of them, LoRA just adds a small, extra part on top of the model – like adding a small add-on instead of rebuilding the whole machine. Only this small add-on needs to be trained, not the whole model. This makes everything much faster and needs way less computing power.
- APO (Automatic Prompt Optimization): Every AI model follows instructions (“prompts”) that tell it what to look for. APO automatically makes these instructions better over time. So instead of retraining the model, the system just gets smarter instructions – which takes much less time to update.
Together, these two methods mean the team doesn’t need to rebuild the whole system every time a new AI model appears. They can just add a small update on top – getting ready to catch new scam patterns in days, instead of taking months to retrain everything from scratch.
What the Results Actually Mean
Over a six-month evaluation period, the paper reports the following results:
These numbers make more sense once you understand what they actually mean.
- 32% faster cluster validation means reviewers could confirm whether a group of accounts was really working together much faster than checking each account by hand. Manually investigating account connections usually takes a lot of time and effort – this cuts that down.
- 50% faster content review shows how much quicker the system could tell if flagged content was AI-made. This is possible because the system already turns each video into a short summary first, so reviewers don’t need to watch the full video every time.
- Under 1% overturn rate means fewer than 1 out of every 100 decisions got reversed later. This is important because it shows the system didn’t sacrifice accuracy just to work faster.
The team also sets a high bar before the system acts on its own. To automatically remove content, it needs to be 92-95% sure. To automatically approve content, it needs to be up to 96% sure. In simple words, the system only acts when it’s very confident – this reduces the chance of wrongly flagging real creators who are just using AI tools normally.
Here’s how the content classifier performed for each type of content, based on the paper’s own results:
| Synthetic Content Category | Decision Type | Precision | Recall |
|---|---|---|---|
| Synthetic Text-to-Speech Narratives (e.g., Racy Stories) | Automate VIOLATES | 95% | 83% |
| Synthetic Text-to-Speech Narratives | Automate APPROVES | 93% | 95% |
| Generative NSFW/Slop (Sex & Nudity) | Automate APPROVES | 88% | 95% |
| Generative NSFW/Slop | Automate VIOLATES | 95% | 68% |
| Procedural Shock/Gore (Child Safety) | Automate APPROVES | 72% | 95% |
What About Legitimate Creators Using AI?
If you use AI tools in your own content, this is probably the question on your mind – and the researchers actually talk about it directly, in a dedicated section of the paper about ethics and responsible AI.
Here’s the concern: a system like this could accidentally punish real creators who are simply using AI to help make their content – not just the people mass-producing spam. The researchers call this risk “definition drift” – where the line between “spam” and “genuine AI-assisted work” starts to blur, and the system ends up being unfair to honest creators.
To prevent this, the system follows one clear rule: it would rather miss some spam than wrongly punish a real creator. In other words, when it’s not fully sure, it chooses to do nothing rather than risk a false accusation.
On top of that, there’s a second safeguard built in. The system only takes action when it sees a group of accounts working together – sharing the same patterns, timing, and templates. If it’s just one person, on one channel, trying out a new AI tool -even if their content looks a little synthetic – the system won’t flag them. It’s specifically built to catch coordinated, mass-production behavior, not individual creators experimenting with AI.
Using AI to help make one piece of content isn’t the target. Running dozens or hundreds of accounts to mass-produce templated, scammy content – that’s the target.
What This Means If You Create Content
For SEOs, publishers, and video creators, this research points to a bigger direction Google is likely to take across all its platforms – not just video, but web content too, eventually. The researchers even mention that the text-detection method they use (called Sentence-BERT, which spots AI-written, scripted content) could work beyond video as well.
The takeaway here isn’t “stop using AI.” It’s this: the target is mass-produced, templated, low-effort AI content – especially content published at an unnatural speed or scale, across many accounts or sites. If you’re a creator or brand genuinely making useful content, even with the help of AI, you’re not who this system is built to catch.
AI-generated spam will likely keep getting smarter as AI tools improve, and attackers will keep finding new ways to create endless versions of the same scam. But the bigger idea in this research goes beyond just one system. Going forward, content moderation is likely to rely less on checking individual posts, and more on spotting coordinated behavior, AI-content patterns, and the networks of accounts behind them.
- It’s about networks, not single videos. S-CTS doesn’t check one video at a time – it looks for groups of accounts working together.
- Two checks have to match. The accounts need to look connected, and the content needs to look AI-made – only then does the system act.
- It’s built to be fast and cheap. The system first turns videos into short summaries, then makes decisions based on that – saving time and cost.
- It can keep up with new AI models. Using LoRA and APO, the system adapts to new tools like Sora or Kling without needing to be rebuilt from scratch.
- Individual creators aren’t the target. The system is designed to catch coordinated, mass-produced content – not someone simply using AI on their own.
AI-generated spam is only going to get harder to spot – but if you’re creating genuine, useful content, that’s not something to worry about. Systems like this are built to notice the difference.
Want to keep your own content ahead of the curve? Check out our latest SEO updates to see what else is changing in how platforms rank and review content.
