Cloudflare has launched a new suite of tools designed to give website owners and content creators more control over how their content is used by artificial intelligence.
The AI Audit tools address the growing concern that bots are scanning websites without permission or compensation for creators.
Ongoing efforts to stomp out AI bots from Cloudflare follow earlier work to identify such unwanted scrapers via digital fingerprinting, designed to differentiate bots from legitimate web users.
Cloudflare wants to put an end to AI bots
Bots themselves are not the issue – those used by search engines can provide value by indexing content and driving traffic to websites. Though AI arguably makes it more challenging to eliminate just certain kinds of bots.
AI bots used by large language models often scrape publicly available data to train models without attributing or crediting sources, and without giving compensation to creators. This can lead to creators finding their work, or similarities, in AI-generated responses.
Cloudflare’s AI Audit was built to give website owners detailed analytics that offer transparency into which AI bots are accessing their sites, how often, and which parts. For example, it can differentiate bots like OpenAI’s GPTBot and Anthropic’s ClaudeBot.
Cloudflare CEO Matthew Prince commented: "Content creators and website owners of all sizes deserve to own and have control over their content. If they don’t, the quality of online information will deteriorate or be locked exclusively behind paywalls.”
As well as identifying bots, Cloudflare’s tools offer controls and custom rules to permit or block AI services to eliminate scraping altogether or to align with deals that website owners may have struck up with certain AI companies.
More from TechRadar Pro
- Check out the best web hosting services
- Cloudflare wants to help stomp out AI scraper bots for good
- Worried about being tracked? Here are the best privacy tools and anonymous browsers
