The use of Artificial Intelligence (AI) bots in the digital publishing industry has been a topic of discussion and analysis in recent times. Data journalist Ben Welsh conducted a study to determine the prevalence of news websites blocking AI bots, such as Applebot-Extended, OpenAI’s bot, and Google-Extended. The results of the analysis shed light on the strategies adopted by news publishers in dealing with these AI agents.
Welsh’s study revealed that approximately 25% of the news websites surveyed were blocking Applebot-Extended, while a higher percentage (53%) were blocking OpenAI’s bot. It was also noted that Google-Extended, introduced last year, was being blocked by almost 43% of the websites. This indicates a significant level of resistance towards AI bots, with a noticeable increase in the number of sites blocking them over time.
Strategic Approaches by News Publishers
According to Welsh, there is a divide among news publishers regarding their stance on blocking AI bots. While some websites choose to block these bots, others enter into partnerships with AI companies, possibly for financial reasons. Companies like OpenAI and Perplexity have secured partnerships with various news outlets and social platforms, leading to speculation about the business strategies involved in such agreements.
There is evidence to suggest that partnerships play a crucial role in determining whether AI bots are allowed access to news websites. For instance, Condé Nast, which previously blocked OpenAI’s web crawlers, unblocked them after announcing a partnership with the company. Similarly, Buzzfeed only allows AI bots that are part of a partnership agreement to crawl its site, highlighting the importance of collaborations in this context.
The Challenges with Blocking AI Bots
With the proliferation of new AI agents and the manual editing required for robots.txt files, maintaining an up-to-date block list can be a challenge for news publishers. Gavin King, founder of Dark Visitors, emphasizes the difficulty faced by website owners in identifying which AI bots to block. This has led to the emergence of services like Dark Visitors, which automate the process of updating robots.txt files for clients, particularly those concerned about copyright issues.
The significance of AI bots in the digital ecosystem has elevated the role of media executives in deciding which bots to block. Some CEOs from major media companies directly oversee this process, indicating the strategic importance of AI bot management. Vox Media’s senior vice president of communications, Lauren Starke, mentioned that they block Applebot-Extended and other AI scraping tools across all properties without commercial agreements.
The analysis of news websites’ approach towards AI bots reveals a complex landscape shaped by partnership agreements, business strategies, and copyright concerns. As the digital publishing industry continues to evolve in the AI age, the decisions made by news publishers regarding AI bot access will have a significant impact on content distribution and data privacy. It is essential for publishers to navigate this dynamic environment strategically and ethically to ensure a sustainable future for journalism in the digital era.
Leave a Reply