This discussion revolves around Cloudflare's new marketplace designed to charge AI bots for scraping websites, with a strong emphasis on the implications for content creators, AI companies, and the future of internet access.
The "Permission-Based Approach" to Web Crawling
A central theme is Cloudflare's push for a "permission-based approach to crawling," which aims to allow website owners to control and monetize access for AI bots. This is supported by the announcement that "Several large publishers, including Conde Nast, TIME, The Associated Press, The Atlantic, ADWEEK, and Fortune, have signed on with Cloudflare to block AI crawlers by default." This initiative is seen by some as a necessary step to address the unchecked data consumption by AI models.
The Problem of Uncompensated Data Consumption and Training
Several users highlight the ongoing issue of AI models being trained on vast amounts of web data without explicit permission or compensation to the content creators. The sentiment is captured by "yantramanav," who questions, "While this is a neat idea, how does it negate all the data theft being done by the bots so far?" The concern is that even with Cloudflare's solution, the data has already been ingested: "I'm afraid the cat's out of the bag now." This is further elaborated by "teruakohatu" noting, "All the LLMs are being trained on LibGen/Anna’s Archive so it’s not in the least surprising they can tell you about papers behind paywalls."
Potential for Bypass and Evasion by AI Companies
A significant point of contention is the likelihood that AI companies will find ways to circumvent Cloudflare's new system. "Toritori12" suggests, "prob will be to cheaper bypass CF considering the amount of data that big techs are consuming." The worry is that these large entities will always find a cost-effective method, especially if "Google Search?" is exempt. "crgwbr" echoes this sentiment, stating, "All this is going to do is drive AI companies to mask their user agent to appear as a standard browser, resulting in a worse end state than we’re in now. It’s an exercise in futility." "rkrisztian2" agrees, asserting, "I agree, it's only the big tech companies who do this AI crawling, and they will always have money for it. This paywall won't stop them."
The Monetization of Content and "Enshittification"
Some users express a pragmatic, albeit cynical, view on monetizing content for AI. "some_furry" joyfully exclaims, ""Read my blog for free, or pay $25/page for your AI to read it for you." This is praxis. Enshittify the enshittification machine." This perspective suggests embracing the trend and profiting from it, even if it means participating in what some perceive as a degradation of the internet's openness. "aspenmayer" questions the value proposition, asking, "Do you think that there is $25 of value in the creation of your blog, to say nothing of value that AI may be able to extract from it?"
The Need for Open Protocols and Legislation
Several contributors argue that a proprietary solution like Cloudflare's is not ideal. "suyash" advocates for "an open source protocol that handles permission and payment for crawlers/scraper," believing it would be a better approach than relying on a single vendor. "delusional" goes further, stating, "This isn't the kind of problem that really ought to be solved through courts. It's obvious to anyone that this is a new kind of problem that no author of the current jurisprudence envisioned. We need new legislation to stop this kind of abuse of the commons." Later, "delusional" reiterates, "We don't need another technical protocol. We need legislation."
Cloudflare's Role as a "Merchant of Record" and Infrastructure Provider
The discussion touches upon Cloudflare's specific role in this new system. "rswail" notes that "CF is acting as the merchant of record, so they will be the ones billing, it's unclear what cut of the price they will take (if any) or if they will include it in their bundled services." There's speculation about expanding this to include "micropayments and subscriptions" and "integration with the browser UI/UX." "asim" points to efforts like Coinbase's "x402" as examples of technical solutions for micropayments using the HTTP 402 status code, suggesting this is a broader trend.
Alternative Solutions: Shared Crawling Infrastructure
A more collaborative and potentially effective solution is proposed by "a_c." This user suggests that AI companies should "collaborate on shared infrastructure" for crawling. The idea is that a "single crawler they all contribute to" would reduce the load on websites and improve compliance with crawling rules. "Instead of all the different companies hitting sites independently, there should be a single crawler they all contribute to." This approach, in their view, is "pretty unimaginative and not the least bit compelling" that Cloudflare instead opted for a direct "pay up" model.
The Erosion of Internet Neutrality and User Experience
Concerns are raised about the broader implications for internet neutrality and the user experience. "greatgib" worries about "the world where neutrality of internet explode... Soon they could decide if your requests come from a specific company IP or networks, because you look suspicious..." They also posit that blocking bots through payment schemes might inadvertently allow "bad actors [to] have a free pass if they pay." "bgwalter" laments the shift where "humans get the Cloudflare captchas in order to access their own content... and 'AI' crawlers get the data highway for a fee." "jgrahamc" clarifies that Cloudflare no longer uses CAPTCHAs broadly, referencing their "Turnstile" product. "koolba" counters by noting that frequent cookie clearing can still lead to such "gate keeping."