Common Crawl Published a Manual for Being Visible to AI, I Automated It

Every month, a non profit called Common Crawl fetches over 2 billion webpages and gives the copy away to anyone who wants it. That free archive is where most AI training data starts.

When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was roughly 60% filtered Common Crawl by training weight. If a model already knows your brand without searching the web, odds are this is part of where it learned.

Where Common Crawl Sits In The AI Pipeline

The crawling has run monthly since 2008, and the snapshots are the raw material for C4, RefinedWeb, RedPajama, Dolma, FineWeb and most of the other big training sets. Labs rarely touch the raw crawl. They take a filtered derivative, so. The strategic issue is whether automated visitors can understand, trust, and complete the same journey a human visitor can. Agent readiness is partly technical, but it is also about clear tasks, accessible flows, and reliable evidence.

The useful check is whether this improves the system behind search performance, not only the words on the page. Internal links, crawlable content, clear entities, current evidence, and a sensible page structure all help the recommendation become easier to trust.

The Audit Common Crawl Expects You To Run

In June, Common Crawl published something interesting. The AI Visibility Audit is an 18 page guide telling site owners how to check their standing in the dataset. You curl your homepage with CCBot's user agent and compare the response to a. The strategic issue is whether automated visitors can understand, trust, and complete the same journey a human visitor can. Agent readiness is partly technical, but it is also about clear tasks, accessible flows, and reliable evidence.

The risk is usually hidden in the execution layer. A page can look fine to a human and still fail for an automated visitor if the form, call to action, rendering path, or confirmation step is not accessible enough for the agent to complete the task.

What The Checker Shows

Type a domain and the diagnosis comes back in four panels. The practical question is what this changes in the system: the page structure, the evidence presented, the measurement habit, or the way the topic is connected to related work.

The practical value is in connecting the idea to an observable signal. That means deciding what should be checked, what would prove the issue is real, and where the team should make the smallest useful improvement first.

Captures Per Crawl

How many of your pages each of the last twelve monthly crawls took, as a trend. My site went from two captured pages in last August's crawl to 55 in July's, which is the most flattering chart anyone has ever drawn of it. A big domain gets. The practical question is what this changes in the system: the page structure, the evidence presented, the measurement habit, or the way the topic is connected to related work.

Robots.txt History

Common Crawl stores the robots.txt it read before every crawl. The checker checks those stored copies backwards month by month and diffs the AI crawler rules, so a block gets a start date. Run bbc.com and every month reads the same. CCBot. The strategic issue is whether automated visitors can understand, trust, and complete the same journey a human visitor can. Agent readiness is partly technical, but it is also about clear tasks, accessible flows, and reliable evidence.

Whose Block It Looks Like

When a block matches a known template, the tool says so. Cloudflare's managed robots.txt has a distinctive license comment. Squarespace's default has its own unique signature. And when eight or more AI tokens go down in one sweep, that's. The practical question is what this changes in the system: the page structure, the evidence presented, the measurement habit, or the way the topic is connected to related work.

A Live Probe

History says what happened, so this one checks what's true right now. It fetches your homepage as CCBot/2.0 next to a normal browser and a control bot, which catches the block robots.txt never shows, a firewall challenging the user agent. The strategic issue is whether automated visitors can understand, trust, and complete the same journey a human visitor can. Agent readiness is partly technical, but it is also about clear tasks, accessible flows, and reliable evidence.

Sitemap Coverage

The sitemap panel is the feature I wanted for myself. The checker fetches your sitemap and diffs it against the captured URLs, which turns the headline number into a work queue. Mine reads 42 of 97 pages in the July crawl, and the other 55. The practical question is what this changes in the system: the page structure, the evidence presented, the measurement habit, or the way the topic is connected to related work.

Tracking A Single Page

You can also paste a full page URL instead of a domain and get that single page's history across the year. Doing that taught me two things about my own site inside a minute. My ChatGPT sources teardown was captured in only 1 of the last 12. The strategic issue is whether automated visitors can understand, trust, and complete the same journey a human visitor can. Agent readiness is partly technical, but it is also about clear tasks, accessible flows, and reliable evidence.

Stored Copy Check

The last piece is a stored copy check. The tool pulls the actual bytes CCBot archived for your homepage, extracts the text, and compares it against your live HTML. Both sides are read before JavaScript runs, which is the point, because. The practical question is what this changes in the system: the page structure, the evidence presented, the measurement habit, or the way the topic is connected to related work.

What the visibility signal actually changes

What the visibility signal actually changes: common Crawl Published a Manual for Being Visible to AI, I Automated It: the Practical Angle should be treated as a visibility signal, not a standalone headline. Introduction Every month, a non profit called Common Crawl fetches over 2 billion webpages and gives the copy away to anyone who wants it. That free archive is where most AI training data starts. When Mozilla audited the LLMs released between 2019 and 2023,. This connects with to Get Cited & Stay Visible when the same signal needs a clearer operating decision. A useful companion note is Working Framework, because it looks at a nearby part of the same system.

What the visibility signal actually changes: the practical question is whether the page, brand evidence, and surrounding content make the answer easier to trust. If that support is weak, search systems can still understand the topic but fail to connect it confidently to the brand. The same pattern also shows up in Working Framework, where the practical question is how the signal becomes visible.

What the visibility signal actually changes: that is why the response should begin with an audit of the evidence already on the site before creating a new asset. The fastest improvement is often a clearer page, a better internal link, or a stronger explanation of why the brand belongs in the answer.

Where the evidence needs to be tested

Where the evidence needs to be tested: a single study or ranking observation should not become a strategy by itself. It should become a diagnostic prompt: which source is being trusted, which query pattern is affected, and which part of the site would make that trust easier to earn?

Where the evidence needs to be tested: that keeps the response grounded. The goal is to improve the evidence chain around the topic rather than publish another summary that repeats what every other page already says.

Where the evidence needs to be tested: the important distinction is between a useful signal and a fashionable talking point. A useful signal changes the brief, the page structure, the linking plan, or the measurement view.