Cloudflare, one of the internet's largest infrastructure providers, is trying to change a basic assumption of how the web works: that AI systems can freely scrape the content they are built on. Under a new policy announced on July 1, the company will push AI firms toward a simple choice, pay for the content they consume or lose access to it. The lever is Cloudflare's position in front of a very large share of the world's websites, which lets it set defaults that individual publishers have struggled to enforce on their own. The framing the company is using is that the old bargain of the web, be crawled in exchange for being sent readers, has broken down in the age of AI, which takes the content but often sends nothing back.
The core of the policy is a demand that AI companies stop hiding behind general purpose crawlers. Cloudflare is requiring them to separate the bots they use for traditional search indexing from the ones they use for AI training and for AI agents, so that a website can keep saying yes to being found on a search engine while saying no, or naming a price, to being used as training data. To give the demand teeth, Cloudflare is changing its own defaults. Starting September 15, 2026, its settings will by default block so called mixed use crawlers, ones that refuse to declare which purpose they serve, from any page that hosts advertising. That default will apply to new Cloudflare customers, to new sites created by existing customers, and to all of the company's existing free tier customers, unless they choose to turn it off.
The second half of the plan is about money rather than blocking. Cloudflare already offered a feature called Pay Per Crawl, which let a website charge AI bots each time they scraped a page. The new step, called Pay Per Use, tries to tie payment to value rather than to raw fetching. Instead of charging only when a page is downloaded, it is meant to compensate a publisher when their content actually does something useful for the AI company, for example when it appears in a generated answer or when it unlocks premium material behind a paywall. Cloudflare named Ceramic.ai and You.com as early partners, and it has also acquired a content licensing marketplace called Human Native to help broker these deals. The company points to its own data that more than half of AI crawler traffic is spent re fetching pages that have not even changed, which it presents as pure waste that a smarter, paid system could eliminate.
What makes this more than another publisher complaint is that Cloudflare is one of the few players with the leverage to make a new default actually hold. Because it sits in front of so much of the web, a change to its out of the box settings quietly rewrites the rules for a large number of sites at once, without each of them having to fight the battle alone. Chief executive Matthew Prince put the reasoning in stark terms, saying that now that the majority of traffic on the internet is non human, the company must go further and act faster so that a sustainable ecosystem can emerge. In other words, the old model where a human visitor might see an ad or a subscription prompt is being hollowed out by machines that read the page and move on, and someone has to redesign the exchange.
The move is not without its own tensions, and they are worth naming. The most obvious is that it concentrates a lot of power in a single company, because when Cloudflare changes a default, it is effectively making policy for a chunk of the internet, and not everyone will be comfortable with a private infrastructure provider acting as the web's tollbooth. There are practical questions too, since measuring when content creates value is far harder than counting when a page is fetched, and large AI companies may resist, route around the rules, or simply refuse to identify their crawlers. It also rolls out unevenly, landing first on free and new customers rather than everyone at once. But the direction is clear and it matters, because it turns a fuzzy, unresolved grievance, that AI is built on other people's work without paying for it, into a concrete mechanism with a date attached. Whether or not this exact system wins, Cloudflare has made the question of who pays for the content behind AI much harder to keep ignoring.