Beginning Sept. 15, all new websites signing up for Cloudflare, as well as all the customers on its free tier, will have the default settings in their bot management protocol set to block “multi-purpose crawlers” on any webpage that has ads. This means that any crawler that scrapes for both search indexing and AI training will be turned away at the door, unless the site owner decides otherwise.

“We’ve been clear about what we want,” said Cloudflare chief strategy officer Stephanie Cohen. “We want a technical solution that allows you to be discoverable without having to give your content away for free.”

While a handful of crawlers fit this description—Apple and Bing, among others—the primary, unnamed target of this action is Google, which infamously uses one crawler to both index sites and train its AI models.

In doing so, Google forces publishers to make an impossible choice: They either allow both functions, enabling Google to scrape their content to train the AI products that are regurgitating their data without compensation; or they shut off both functions and disappear from Google Search, presumably losing their largest source of traffic in the process.

Nonetheless, that publishers are considering opting out of Google Search marks a symbolic milestone in the lifecycle of the open web. The posturing is all part of a broader negotiation, to be sure, but the threats are no longer empty.

No publisher wants to make do without Google, but many are increasingly prepared to do so. Google, on the other hand, has no such backup plan.

adweek.com - Publishers opt out google search