Google Explains 503s, Robots.txt Limits & Bracket URLs, SEO Pulse

Technical SEO often feels like a game of nuance. A single character in a URL or a specific HTTP status code during a ten minute maintenance window can change how Google perceives the health and structure of your site. When these details are overlooked, you risk unnecessary indexing issues or accidental ranking drops.

Recent updates from Google and industry partners highlight where the boundaries lie between crawling and usage, and how the search engine handles modern image formats and URL encoding. Understanding these distinctions allows you to maintain site stability without confusing the bots.

Handling Downtime During PHP Patches

When you take a site offline for PHP patches, the goal is to tell Google that the absence of your content is temporary. Google recommends serving a 503 Service Unavailable response during these windows. This signals that the server is currently unable to handle the request but will be back shortly, preventing the search engine from thinking the page has been permanently deleted or broken.

The technical challenge here is that if your PHP is down, your PHP cannot serve the 503 error. To implement this correctly, the response must be triggered by a layer that remains active while the backend is offline. This usually means configuring a routing rule at the CDN level or pointing your DNS to a temporary server that can handle the request and return the 503 status.

One critical warning from Google is to avoid triggering a 503 in response to robots.txt requests. Doing so can lead to a total block of all crawling activity, which is likely not the intended result of a brief patch window. A useful companion note is Google Explains, because it looks at a nearby part of the same system.

Expert Interpretation: The tradeoff here is between simplicity and safety. While it is easier to just let the site time out or return a 404, the risk is that Google may begin to de index pages if it perceives a prolonged outage. You should inspect your CDN capabilities to see if you can automate a "maintenance mode" that serves a 503 without requiring the origin server to be online.

The Boundary Between Crawling and AI Training

There is a common misconception that blocking a crawler in robots.txt prevents that content from being used for AI training. Gary Illyes has clarified that according to RFC 9309, standard robots.txt directives only control what compliant crawlers are allowed to crawl. They do not dictate what happens to the data after it has been collected. The same pattern also shows up in Google Says, where the practical question is how the signal becomes visible.

Essentially, robots.txt is an access preference protocol. If you want to stop a specific AI crawler, you must target its specific user agent token. For those concerned with Google's AI, the Google Extended token is the primary tool. This token allows you to manage whether your content is used to train future Gemini models or for grounding in Vertex AI and Gemini Apps, but it does not affect whether your site appears in Google Search results.

Google Extended is not a separate crawler with its own HTTP user agent string. While there are drafts within the Internet Engineering Task Force to create a specific Content Usage rule for robots.txt to handle training preferences more cleanly, these are not yet approved or implemented.

Expert Interpretation: This creates a significant gap in control. You can stop a bot from visiting a path, but you cannot easily retroactively stop the use of data already crawled unless you use specific tokens like Google Extended. The decision you need to make is whether you prioritize visibility in search (which requires allowing the crawler) or data privacy for AI (which requires the Extended token).

Expanding the Reach of ChatGPT Search

The landscape of AI discovery is shifting. A recent research pilot between Cloudflare and OpenAI has reportedly enabled OpenAI to discover and index hundreds of millions of new web pages. This expansion provides ChatGPT Search with a significantly larger pool of information to draw from when generating responses.

Expert Interpretation: This indicates that AI agents are finding ways to discover content that may have previously been invisible to them. For site owners, this means the "dark web" of unlinked or hard to find pages is shrinking. You should evaluate if your site has legacy content that is public but not linked, as it may now be surfaced by AI agents via these new discovery mechanisms.

The Nuance of Raw vs. Encoded Brackets

URL structure is often treated as a formality, but John Mueller recently clarified a point regarding square brackets. In the eyes of Google, a raw square bracket in a URL and its percent encoded versions (%5B and %5D) are treated as different URLs.

This is based on RFC 3986, the URI syntax standard, which reserves brackets and plus signs. If a website serves the same content for both the raw and the encoded version of a URL, Google will crawl both, identify them as duplicates, and then select one to be the canonical version.

The recommended approach is consistency. You should pick one format and use it everywhere: in your internal links, sitemap entries, canonical tags, and structured data. While Mueller noted that sites already using raw brackets consistently will likely continue to function, he advised against setting up new sites this way, as reserved characters should be percent encoded.

Expert Interpretation: Mixing these formats creates "canonical noise." When Google has to guess which version of a URL is the primary one, you lose a small amount of control over your indexing. If you are auditing a site and see a mix of %5B and raw brackets, the priority should be to standardize them to the encoded version to align with official documentation.

JPEG XL Support in Search and Discover

While JPEG XL has seen adoption in browsers like Chrome and Firefox, it is not yet supported in Google Images or as preview images in Google Discover. Google's official image SEO documentation continues to list BMP, GIF, JPEG, PNG, WebP, SVG, and AVIF as the supported formats for the src attribute of an img element.

To use JPEG XL without losing search visibility, the best approach is to use the picture element. By providing a JPEG XL source within the picture element but keeping a supported format (like WebP or JPEG) in the nested img element, you ensure that search engines can still read and index the image via the src attribute.

for those focusing on Discover, it is vital that the og:image and schema.org markup point to one of the supported formats rather than a JPEG XL file, as Discover relies on these for its previews.

Expert Interpretation: This is a classic case of browser support outpacing search engine support. The tradeoff is between cutting edge image compression and visibility. If you move to JPEG XL, you must implement a fallback strategy. Inspect your image pipeline to ensure that your metadata and Open Graph tags are not accidentally serving unsupported formats to bots.

The Future of AI Agents and Web Protocols

Google CEO Sundar Pichai has suggested that AI agents will eventually access the web through two parallel paths: browsing pages like humans do and using specialized protocols designed for agents.

Pichai pointed toward the development of agent and payment protocols, such as the Universal Commerce Protocol. While he believes protocols are a more scalable approach for the long term, they will take longer to develop and standardize. In the meantime, agents will continue to browse the web traditionally.

This has practical implications for site owners. Google's own hosted agents, which browse as Google Agent, are categorized as user triggered fetchers. According to Google, these fetchers generally ignore robots.txt rules because they are acting on behalf of a specific user request rather than performing a general crawl.

Expert Interpretation: We are moving toward a world where robots.txt is no longer the "master switch" for all automated traffic. The distinction between a "crawler" (which indexes for a database) and an "agent" (which fetches for a user) is becoming critical. You should recognize that blocking a user agent in robots.txt may stop your site from being indexed, but it may not stop an AI agent from accessing a page if a user explicitly asks it to.

Understanding the Signals Sent to Crawlers

When looking at these updates together, a clear theme emerges: the importance of the specific signal you send to a bot. Each technical choice communicates a different intent to the search engine.

A 503 status code is a signal of temporary unavailability, which protects your rankings during maintenance. A Disallow directive in robots.txt is a signal to compliant crawlers to stay away from certain paths, but it is not a legal or technical barrier to how that data is used for AI training unless combined with tokens like Google Extended.

Similarly, the way you handle brackets in your URLs signals whether you have a clean, single source of truth structure or a fragmented one. The choice of image format in your img tags signals to Google Images whether a file is "visible" or "invisible." Finally, the shift toward AI agents suggests that the signals we use to control bots are evolving from simple "keep out" signs to more complex protocols of interaction.

The common thread is that precision matters. Whether it is the difference between a raw bracket and an encoded one, or the difference between a crawler and a user triggered agent, the details determine how your site is discovered, indexed, and utilized by the next generation of search technology.