Thanks to a new AI deal, Google is now the only search engine that can scrape Reddit to show new posts in its search results. If competitors like Bing or DuckDuckGo want their users to see the same, they'll have to pay up.
That's according to a new report from 404 Media, which first spotted that search engines other than Google aren't showing any Reddit results from the last week in their search results. If, like me, you frequently tag on "Reddit" or "site:reddit.com" to your search queries, you'll find the results are rather disappointing on Bing or DuckDuckGo right now. You'll either see outdated Reddit posts or a link that reads “We would like to show you a description here but the site won't allow us.”
Reddit spokesperson Tim Rathschmidt told the Verge this development was "not at all related to our recent partnership with Google," referring to a $60 million deal struck in February to let the company train its AI models on Reddit's content.
“We have been in discussions with multiple search engines," Reddit spokesperson Tim Rathschmidt told the Verge. "We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
So rather than any kind of exclusivity agreement, it seems tied to Reddit's recent crackdown on web crawlers. Earlier this month, to enforce its policy against scraping, Reddit tweaked the site’s robots.txt file, which directs AI crawlers to which pages they can or can't glean information from. By preventing certain search engines from scraping its pages with AI bots, Reddit is essentially walling off its content to keep it out of search results. In short, Reddit is taking its ball and going home. Unless you want to play by its rules, that is, which Google, understandably, seems keen to.
Microsoft spokesperson Caitlin Roulston confirmed to the Verge that Bing stopped hoovering up content from Reddit after it updated its robots.txt file on July 1st. “Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models,” she added.
So for now, Google is the exclusive home to Reddit content in its search results. At least until Reddit strikes an agreement with its competitors.
