Why Common Crawl Is Becoming a Publisher Rights Flashpoint

Digital Content Next, a trade body representing US digital publishers, has sent a cease and desist letter to the Common Crawl Foundation. The letter demands Common Crawl stop collecting publisher content and remove material already in its datasets.

DCN CEO Jason Kint announced the legal notice in a blog post, and Press Gazette reported additional details from the letter this week. Common Crawl has crawled several billion new pages each month since 2007 to build a free public archive.

What DCN Demands

DCN claims Common Crawl has "flagrantly infringed" copyrighted content by creating its datasets and sharing them with AI companies. The letter argues "copyright law is not an opt out regime." In other words, DCN's position is that publishers shouldn't have to ask to be excluded.

Common Crawl should need permission to include them. "challenges a growing assumption that content created through substantial investment can be collected, stored, repurposed, and monetized simply because it is technically accessible."

What DCN Demands: this is no longer only a scraping debate. The real issue is whether automated access creates enough discovery, citation, or business value to justify the server cost, analytics noise, and operational risk it introduces.

What DCN Demands: a sensible policy should classify crawlers by value. Search indexing, useful AI referral paths, unknown scrapers, and abusive traffic do not deserve the same rules, and server logs should guide that decision before broad blocking.

Why DCN Doubts The Removal Process

The DCN letter questions whether Common Crawl follows opt out instructions and whether it removes content when asked. Per Press Gazette, DCN's lawyers are examining whether Common Crawl's statements to publishers "may have been inaccurate or misleading." Common Crawl publishes a public registry of websites that have asked not to be scraped. A useful companion note is Personalization Can Help Small Publishers, because it looks at a nearby part of the same system.

It includes entries for the Associated Press, the BBC, and a large News/Media Alliance submission covering hundreds of domains. Press Gazette reports the list also includes other major publishers. This isn't the first time the removal process has been questioned.

Why DCN Doubts The Removal Process: the operating question is crawler value. Some automated visits support discovery, but others consume infrastructure, distort logs, and create no visible demand signal for the business.

Why DCN Doubts The Removal Process: the decision should start with evidence from logs, not frustration alone. Crawl rate, requested paths, referrers, user agents, and server impact show which bots deserve access and which do not.

Common Crawl's Response

Common Crawl executive director Rich Skrenta declined to comment on the letter when contacted by Press Gazette. He has pushed back on similar claims before.

In a November blog post responding to The Atlantic, Skrenta denied that the organization lied to publishers or scrapes paywalled material. He said the archive's file format can't be edited after publication without breaking its integrity. Instead, Common Crawl says it removes or filters affected URLs from subsequent crawls and makes them inaccessible through its public tools and indices: "When a publisher asks us to remove previously crawled material, we respond promptly and initiate a removal process that reflects the technical design of our dataset." "No one at Common Crawl has ever claimed this work was instantaneous or complete; rather, we have been open about its complexity and ongoing nature." In a forum post this week, Skrenta said Common Crawl is contributing to open standards work on how websites express AI scraping preferences.

Common Crawl's Response: a useful bot policy needs more nuance than allow or block. Search indexing, AI referral paths, training crawlers, and unknown scrapers should be evaluated against different business outcomes.

Common Crawl's Response: the practical policy is graduated control: allow what creates discoverability, throttle what is expensive, and block what shows no legitimate value.

Why this changes the operating question

The DCN letter targets the stored archive, not just future crawling, and argues the burden should not fall on publishers to opt out in the first place. Most publishers in BuzzStream's sample have already made the blocking decision, with 79% of the 100 news sites it checked blocking at least one training bot.

Cloudflare's Year in Review data we covered in January found CCBot among the bots with the most full disallow directives across top domains. The question DCN raises is what those blocks accomplish if years of content stay available for training anyway.

Why this changes the operating question: the cost side matters because server load is not abstract. If bot activity slows pages, inflates analytics, or forces infrastructure spend, the visibility benefit has to be proven more carefully.

What to watch next

Whether DCN escalates depends on how Common Crawl responds, and Common Crawl hasn't said how it will. The two sides want different rules for who acts first.

Skrenta is backing standards work that would let sites state their scraping preferences, which keeps opting out as the model. The UK's CMA took a similar path when it required Google to let publishers opt out of AI search features. DCN argues scrapers should need permission first.

What What DCN Demands changes in practice

What What DCN Demands changes in practice should be checked against the page and the wider search system, not treated as an isolated note. The goal is to find the weakest missing proof point and improve that before expanding the topic further.

Where Why DCN Doubts The Removal Process needs stronger evidence

Where Why DCN Doubts The Removal Process needs stronger evidence is useful only if it changes a real operating habit. That could mean updating the page structure, strengthening entity evidence, improving a profile, changing a reporting view, or clarifying the path from answer to action. This connects with What Gemini Business Profile Tools Mean when the same signal needs a clearer operating decision. The same pattern also shows up in Why Real Experience Still Separates SEO Content, where the practical question is how the signal becomes visible.

Where Why DCN Doubts The Removal Process needs stronger evidence: the practical policy is graduated control: allow what creates discoverability, throttle what is expensive, and block what shows no legitimate value.

What should be checked before changing the strategy

What should be checked before changing the strategy: The first check is whether the page already answers the question with enough specificity. If the visible content, internal links, and supporting proof are thin, publishing more around the same topic will only spread the weakness across more URLs.

Why Common Crawl Is Becoming a Publisher Rights Flashpoint should therefore be treated as part of an operating system, not a one off news item. The useful outcome is a clearer page, a cleaner signal, and a better decision about what deserves attention next.

Where the signal needs stronger proof

Where the signal needs stronger proof: The evidence layer matters because search systems need corroboration. A strong page should connect the entity, the topic, the offer, and the user's intent without forcing a crawler or an assistant to infer the missing pieces.

How to avoid turning the story into a generic checklist

How to avoid turning the story into a generic checklist: The risk with a fast response is that the team turns every new signal into another checklist. The better habit is to ask which part of the system changed and whether the change is large enough to justify new work.

What this means for the next content refresh

What this means for the next content refresh: A useful refresh should improve the reader's decision, not only update the date. That can mean adding clearer examples, tightening the page structure, linking to related posts, or removing claims that no longer match the current search environment.