Policy & Regulation
Patreon moves from asking AI bots not to scrape to blocking them
Patreon is working with Cloudflare to actively block AI bots that scrape creators' content for AI training, moving beyond its earlier robots.txt requests after weekly scraping attempts by individual crawlers dropped from thousands to zero in testing.
Patreon, the membership platform for creators, is stepping up its defenses against AI companies scraping its content to train models. The company said Thursday it is working with internet infrastructure provider Cloudflare to directly block AI bots designed to train on creators’ work without permission, using Cloudflare’s AI Crawl Control technology.
Patreon first put measures in place to deter AI crawlers in 2023, but the company says scraping has grown more sophisticated since then. Its paywall has long kept much of creators’ content out of crawlers’ reach, but newer discovery tools — a redesigned Home Feed and a tweet-like feature called Quips — could expose more content to scraping, prompting the stronger response.
The key shift is enforcement. Rather than only asking AI crawlers not to scrape via robots.txt — a standard file that tells bots how they may use a site, but one crawlers can simply ignore — Patreon is now actively blocking AI training bots. “Consent shouldn’t depend on whether a scraper chooses to behave,” a Patreon blog post said of the stricter measures. When Patreon tested the new blocking features, individual AI training crawlers’ weekly attempts to access the site went from “thousands of attempts to zero,” according to the post — evidence, Patreon says, that the crawlers had been ignoring its robots.txt instructions and scraping anyway.
Patreon said it will still allow bots that index pages and organize information in ways that send users back to the platform, distinguishing them from bots built purely to train AI models.
The move follows a broader shift among online publishers grappling with AI ingesting their work. Cloudflare now offers tools letting website publishers restrict AI bots, including a marketplace called Pay Per Crawl that lets sites charge AI bots for scraping. Earlier this month, Cloudflare changed its policies so that “mixed-use” crawlers — those that both index a site for search and train AI on its content — are blocked by default on any page that hosts ads.
“As AI agents become increasingly powerful and popular, creators deserve a meaningful say in how their work is used by AI companies,” Patreon product chief Drew Rowny said in the announcement. “On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used.”
Why it matters
Patreon’s move signals that creator platforms are done relying on the honor system of robots.txt and are instead using infrastructure-level blocking to enforce consent over AI training — a template other publishers weighing Cloudflare’s new crawler-blocking tools may follow.