Get all your news in one place.
100’s of premium titles.
One app.
Start reading
PC Gamer
PC Gamer
James Bentley

Cloudflare is allowing websites to block AI from scraping them and can even make bots pay for access

A picture of the PC Gamer website.

If you own or run a website and don't want AI bots crawling through your work to train its dataset—and who does?—Cloudflare has launched the ability to "block all AI bots in one click", with even more interesting features coming down the pipeline in the future. 

According to a new Cloudflare blog, as spotted by Ars Technica, the cloud-based content delivery and management service has enabled a whole host of tools to better manage concerns around AI from the sites it supports. 

The new AI functions, labelled "AI Audit", are split into three main actions. The first is that Cloudflare users can control bots' access to websites. In the blog post, it said: "Many small sites don’t have the skills or bandwidth to manually block AI bots. The ability to block all AI bots in one click puts content creators back in control." 

This new system can not only do a better job at differentiating between 'good' and 'bad' uses of AI bots but also gives site owners the ability to curate which bots they block. Secondly, site owners now have a better feed of analytics from AI, like being able to spot how many are crawling a site and which uses of AI source their page.

As a result of these in-depth analytics, users can better negotiate with AI bot owners for access to the site. 

Finally, though this isn't live with the rest of the tools, website owners can now directly sell access to AI crawlers. It can set prices for those accessing the site en masse, and get paid when they are cited or trained on. The first two actions can be monitored right now if you use Cloudflare, but the price setting is set to be supported at some point in the future, with the ability to sign up to a waiting list now.

AI, explained
(Image credit: Jakub Porzycki/NurPhoto via Getty Images)

What is artificial general intelligence?: We dive into the lingo of AI and what the terms actually mean.

The Cloudflare CEO and co-founder Matthew Prince makes an interesting point in the blog for the inclusion of these AI monitoring tools. As sites are increasingly worried about their work being lifted without credit, and then boosted above them in search results, Prince talks about the effects this can have on both the reader and creator.

"Content creators and website owners of all sizes deserve to own and have control over their content," Prince says. "If they don’t, the quality of online information will deteriorate or be locked exclusively behind paywalls."

If only AI bots are paying for those paywalls, it might disincentivise the land grab effects of mass generative AI theft on the online landscape. AI Audit has only just rolled out but it could offer better protections for site owners, especially when the price-setting tool arrives.

Sign up to read this article
Read news from 100’s of titles, curated specifically for you.
Already a member? Sign in here
Related Stories
Top stories on inkl right now
One subscription that gives you access to news from hundreds of sites
Already a member? Sign in here
Our Picks
Fourteen days free
Download the app
One app. One membership.
100+ trusted global sources.