What to Check When Cloudflare Marks a Verified Bot
Treat Cloudflare’s Verified Bot label as an identity signal, not proof of good behavior. For a WordPress site, check the live robots.txt response against the crawler preferences you intended to publish, inspect any separate Cloudflare blocking controls, and review available traffic records before deciding what to change. A robots.txt directive expresses a preference; it does not technically prevent a crawler from accessing content, as Cloudflare explains in its robots.txt Setting documentation.
What “verified” does—and doesn’t—tell you
Cloudflare’s October 16, 2026, accountability announcement says a bot can be verified and still ignore preferences, overload an origin, or evade controls. Cloudflare also notes that misconfiguration or bugs may explain behavior. Its system is being used internally to assess sustained patterns, including how a bot responds to HTTP 429; customer-selectable trust thresholds are planned, not described as a generally available customer control. The announcement does not establish that site owners can see Cloudflare’s internal evaluation or a per-bot compliance history.
Cloudflare’s Web Bot Auth documentation describes cryptographic signatures in HTTP messages as one method of verifying that a request comes from an automated bot. That helps explain the distinction: verifying identity does not establish what the bot will do next, whether it follows a site’s published preferences, or whether it is appropriate for your site to allow.
1. Inspect the live robots.txt response
Open https://example.com/robots.txt, replacing example.com with your site’s domain. Record the date and time, the final URL after any redirect, the HTTP response code, and the directives in the response body. If you use a browser, its developer tools can help you confirm the response status; a request tool or hosting log may provide additional detail if available. Don’t assume the file is present, returns a particular status, or contains the same directives as a Cloudflare settings screen.
Compare the response with the policy you actually intend to publish, including which crawler categories or user agents the directives address. Cloudflare’s robots.txt Setting documentation explains that live content can depend on zone settings, the origin file, and policy choices, so the response from your own site is the evidence to check. A mismatch means the published preference needs investigation; it does not, by itself, establish that a crawler ignored it.
2. Check preference sync, then verify the file again
If you expect Cloudflare-managed directives, check whether Bot Preference Sync is enabled and whether at least one relevant Search, Agent, or Training policy is set to a blocking or disallow option. Cloudflare documents that those conditions are required for Bot Preference Sync to add the managed directives. After changing a setting, reload the live robots.txt response and record the new status, timestamp, and content. The test is whether the response now expresses the preference you intended—not whether a crawler has obeyed it.
Hypothetical store example: A store intends to disallow a crawler category, but its live robots.txt does not show the expected directive. Before changing edge rules, the operator checks whether Bot Preference Sync is enabled and whether the matching policy is set to a blocking or disallow choice. They reload the file and confirm whether the directive appears. If the published preference is still wrong, they continue investigating the file and settings rather than assuming that an enforcement rule will fix its contents.
3. Verify enforcement separately
A published robots.txt preference and an edge block are different things. Cloudflare says robots.txt compliance is voluntary; the file does not technically prevent access. If your goal is to block selected AI crawlers, inspect the relevant AI Crawl Control setting and its policy or rule configuration separately. Check what action is selected and whether the configuration applies to the crawler category and traffic you mean to control. Controls and available details can vary by account, so verify what your own Cloudflare zone exposes. Do not use the presence or absence of a robots.txt directive as proof that a request was blocked.
4. Review Bot Analytics with its limits in view
If available in your account, Cloudflare places Bot Analytics at Security > Analytics > Bot analysis. Check your plan access first: Cloudflare’s Bot Analytics documentation describes reduced-functionality traffic inspection for Business and Enterprise customers without Bot Management, and additional analytics for Enterprise customers with Bot Management. Then note the time range, the traffic type and request attributes the interface makes available, and the sample-rate notice shown in the dashboard. Cloudflare says displayed data may be sampled; don’t treat a sampled count as a complete request total.
Use the analytics to guide a focused investigation, not to infer more than the data shows. If you see a traffic pattern that concerns you, compare it with any relevant records your account exposes and keep the time range consistent. A correlation between crawler traffic and origin load or a business outcome does not, on its own, establish that the crawler caused it.
5. Treat HTTP 429 as a signal, not a complete diagnosis
Cloudflare’s accountability announcement says its internal evaluation considers whether a bot slows down, pauses, or continues after receiving HTTP 429, which Cloudflare describes as a signal to reduce request rate. That explains why the response may matter to Cloudflare’s assessment; it does not mean you can see that assessment in your account. Nor does a 429 record alone identify which bot received it or prove how that bot responded. Attribute a 429 and later requests to a particular Verified Bot only when available records actually link them.
If your plan or server setup gives you relevant request records, compare the available identifiers and timestamps across the 429 response and subsequent requests. If the records do not establish that they belong to the same crawler, record the uncertainty and avoid treating the sequence as proof. For help putting this kind of review into ongoing WordPress operations, Splinternet Marketing’s website maintenance service can help configure and document controls, review available evidence, and keep the checks maintainable. Brian Bateman brings 30 years of experience to the work; the service is $421 per month per business, month to month.
Verification checklist
- Record the live robots.txt response, including its status, final URL, timestamp, and directives.
- Compare the response with your intended crawler preferences and the relevant Cloudflare policy settings.
- If using Bot Preference Sync, confirm it is enabled and the relevant Search, Agent, or Training policy is set to blocking or disallow; reload the live file to verify the result.
- Check the applicable AI Crawl Control or other relevant blocking configuration separately from robots.txt.
- Confirm whether your plan provides Bot Analytics, what traffic details are visible, and the sample-rate notice for the selected time range.
- For a 429 investigation, use only records that link the response and later requests to the same crawler; don’t infer identity or behavior from a 429 alone.
If your live file and Cloudflare settings disagree, which layer did you find was responsible—the preference-sync configuration, the origin file, or an enforcement setting? You can share the general finding without posting private logs or zone details.
Sources
Editorial note: AI assists with research, drafting and automated checks. Sources are linked so you can verify the guidance. Platform requirements can change; confirm the details that apply to your setup.
