Is Cloudflare Blocking Crawlers? Audit AI Crawl Control

Yes, Cloudflare may be blocking a crawler—but an AI Crawl Control Allow or Block setting does not prove what the crawler actually received. The effective result can be changed by WAF rule order, bot controls, redirects, Transform Rules, Workers, robots.txt, origin security or an application error. Audit the complete request path before concluding that AI Crawl Control caused the failure.

This guide is for a small-business site operator or developer responsible for Cloudflare. It keeps Googlebot separate from AI crawler management because a crawler shown in Cloudflare’s AI Crawl Control table is not automatically evidence about Google Search crawling or indexing.

What AI Crawl Control does—and does not—prove

Cloudflare’s AI Crawl Control documentation describes visibility into AI-service crawler activity, per-crawler Allow or Block policies and robots.txt compliance monitoring. The feature is available on all Cloudflare plans. Pay-per-crawl is described separately as a private or closed beta, so do not treat it as part of the ordinary Allow or Block workflow.

When you block a crawler through AI Crawl Control, Cloudflare creates or updates a WAF custom rule. That rule is important, but it is not necessarily the whole policy record. A direct edit to the WAF rule can add hostname, path or user-agent exceptions without updating the AI Crawl Control dashboard. The dashboard and WAF configuration can therefore drift apart.

Cloudflare also documents in its WAF integration guidance that upstream WAF rules can block a crawler selected as Allow. Conversely, Skip, redirect or Transform Rules can allow a crawler selected as Block to bypass the AI Crawl Control rule. The result depends on the expressions and ordering on your zone. Cloudflare’s documented product precedence does not describe every hosting firewall, origin rule, Worker, plugin or external security service.

Cloudflare’s bot-precedence documentation states that WAF custom rules and AI Crawl Control crawler blocks execute before Cloudflare bot solutions, while pay-per-crawl executes after bot solutions. Treat that as guidance for the listed Cloudflare products and configured rules—not as proof that an origin or third-party control cannot intervene.

The weekly audit worksheet

Before changing anything, create one row for each crawler and hostname that matters. For example:

Field	Record
Hostname	Exact public hostname, including whether it is www or non-www
Crawler	Operator name and the exact user-agent documented by that operator
Intended action	Allow, Block or an explicitly limited path policy
AI Crawl Control setting	Current dashboard action and date captured
WAF rule	Generated rule name, expression, action and position
Other controls	Bot rules, Skip rules, redirects, Transform Rules, Workers and origin security
robots.txt	Status, redirect chain and relevant User-agent or Disallow directives
Expected response	Configured block response or normal application response
Last change	Rule or robots.txt change time, if known

Save a screenshot or export of the AI Crawl Control settings and the complete custom WAF rule order before editing. This gives you a comparison point if a later change affects unrelated visitors or legitimate crawlers.

Step 1: Inspect Cloudflare’s enforcement layers

Open the Crawlers table. Record allowed requests and unsuccessful requests for the crawler and hostname. Cloudflare says an unsuccessful request can result from any rule or response error, not only an AI Crawl Control block. Use the metric as an investigation signal, not a diagnosis.
Open the crawler policy. Confirm whether the current dashboard action matches the written intended policy. Note whether the policy applies broadly or has path or hostname exceptions.
Find the generated WAF rule. Review its expression, action, response behavior and position among custom WAF rules. Cloudflare’s Manage AI Crawlers documentation explains the dashboard workflow and generated-rule relationship.
Review rules before and after it. Look for country blocks, rate limits, bot challenges, user-agent conditions, Skip rules, redirects and rules that change request handling. A rule before the AI Crawl Control rule may block an intended Allow. A bypass or redirect after it may change an intended Block.
Check Transform Rules and Workers. Confirm whether the request is rewritten, redirected or handled by a Worker before it reaches the application. A successful Cloudflare response does not necessarily mean the origin served the requested page.
Check origin security. Review hosting-firewall events, web-server access logs and application logs. Cloudflare’s dashboard cannot prove that a request reached the origin.

If an upstream rule is allowing a crawler that should be blocked, Cloudflare recommends moving the AI Crawl Control WAF rule to the top of the custom WAF rules. Do not make that change casually: rule order can affect other traffic. Test the site’s broader security policy after any reorder.

Step 2: Check robots.txt separately

Request the public file directly at your site’s /robots.txt path. Record:

HTTP status code;
every redirect and its destination;
response headers, including cache indicators;
the response body and its current User-agent, Allow and Disallow directives; and
whether Cloudflare and the origin recorded the request.

Cloudflare’s Directives tab can show robots.txt availability, unsuccessful requests, content signals and crawler violations across hostnames. Open the AI Crawl Control Directives documentation when interpreting those fields.

A repeated unsuccessful robots.txt request or a 404 can indicate that crawlers cannot access the file. Check upstream WAF and other security settings rather than assuming the crawler ignored your directives. Also review timing: Cloudflare says violations are not logged in real time, and a Disallow rule added later can make earlier legitimate requests appear as violations.

An inaccessible robots.txt is a delivery problem independent of whether your business wants to Allow or Block a particular AI crawler. Fix availability first, then decide whether the directives match the intended policy.

Step 3: Run one controlled request test

Choose one public, cacheable URL that should be accessible and test its path and /robots.txt separately. Use the exact documented user-agent for the crawler being evaluated. Do not invent a universal crawler user-agent or assume that a user-agent string proves identity.

Set SITE to the site being tested and replace the placeholder user-agent with the operator’s documented value:

curl -sS -D - -o /tmp/crawler-body \
  -A 'DOCUMENTED-CRAWLER-USER-AGENT' \
  -w '\nfinal_status=%{http_code}\nredirects=%{num_redirects}\nurl=%{url_effective}\n' \
  -L "$SITE/public-page/"

Repeat the test for the same public page path and $SITE/robots.txt. Record the exact command time, source location, user-agent, status code, redirect chain, response headers, cache indicators and response body. Then compare the request with Cloudflare request logs and origin access logs.

If Cloudflare logged the request but the origin did not, a Cloudflare-layer action, cache response or Worker may have ended or changed the request. If the origin received it and returned an error, investigate the application, hosting firewall or server response. These tests are diagnostic snapshots; they do not prove that every crawler request from every IP, location or cache state will behave identically.

Decision aid: policy mismatch or delivery failure?

Evidence	Likely interpretation	Next check
Intended Block, and the configured Cloudflare block response appears	The policy may be working for this request	Confirm the responsible WAF rule, request classification and logs
Intended Block, but a normal page or redirect appears	A bypass, exception, Transform Rule or Worker may be intervening	Review rule order, expressions and redirect handling
Intended Allow, but a challenge, block or error appears	Another WAF, bot, origin or security control may be responsible	Compare preceding rules, response details and origin receipt
Unsuccessful request count increases	There is a failed response, but its cause is not established	Inspect the individual request and associated logs
robots.txt is 404, blocked or repeatedly unsuccessful	The crawler may be unable to retrieve policy instructions	Check WAF, redirects, Workers, hosting and the file itself
Directives tab shows a historical violation	The calculation may reflect a later robots.txt rule	Compare violation timing with the rule-change history

Hypothetical example: an allowed crawler meets a country block

Hypothetical scenario: A small U.S. home-service business selects Allow for an AI crawler because the operator wants that crawler to access public service pages. The Crawlers table later shows unsuccessful requests.

Do not conclude that AI Crawl Control blocked the crawler. The developer should verify all of the following:

the AI Crawl Control setting is still Allow for the correct hostname;
the generated WAF rule and its position in the rule order;
the country-block rule expression and its position relative to the crawler rule;
the country Cloudflare assigned to the representative request;
the crawler classification and the exact user-agent used;
the returned status, headers, body and any challenge or block response; and
whether the origin received the request.

If the country rule matches first and Cloudflare returns a block response without an origin request, the evidence supports a WAF conflict—not a conclusion that the AI Crawl Control Allow setting failed. The operator can then decide whether to narrow the country rule, create a documented exception or retain the block. Test the resulting policy against ordinary U.S. visitors and the site’s other security requirements.

Keep Googlebot out of the AI-crawler assumption

Googlebot is a separate Google Search crawler. Google explains in its Googlebot documentation that its user-agent can be spoofed and recommends verification through reverse DNS or Google’s published IP ranges; a user-agent-only test is insufficient.

For important pages, inspect the URL in Google Search Console and review the site’s Google-specific robots.txt rules. Google’s crawling documentation says Googlebot reads robots.txt before requesting a page or resource. Google also distinguishes crawl blocking from index removal: a robots.txt block does not reliably remove a URL from Google’s index, while a noindex directive requires the page to remain crawlable. That is Google Search guidance and should not be generalized to every AI crawler.

AI Crawl Control settings alone therefore do not establish whether Googlebot can crawl, render or index a URL. Confirm the actual Googlebot request, response and Search Console state separately.

Change-control checklist for this week

Save the pre-change AI Crawl Control policy, robots.txt response and complete WAF rule order.
Write the expected response for one intended-allowed crawler and one intended-blocked crawler.
Change one control at a time: do not reorder WAF rules, edit robots.txt and modify a Worker in the same deployment.
Retest the representative public URL and /robots.txt with the recorded user-agent.
Compare status, redirects, headers, body, cache indicators, Cloudflare logs and origin receipt.
Verify that the intended Block still produces its configured block response.
Verify that the intended Allow produces the expected application response unless another documented control intervenes.
Record expected versus observed results, the rule changed, the timestamp and the person who approved the change.

The useful question is not simply “Does AI Crawl Control say Allow or Block?” It is “Which control handled this representative request, what response did it produce, and did the request reach the origin?” That distinction prevents an unsuccessful-request metric or a dashboard setting from being mistaken for a complete crawl diagnosis.

Have you found a Cloudflare crawler failure caused by WAF order, robots.txt delivery or an origin control rather than the AI Crawl Control setting itself? Share the diagnostic evidence without including private logs or business-sensitive information.

Sources

Editorial note: AI assists with research, drafting and automated checks. Sources are linked so you can verify the guidance. Platform requirements can change; confirm the details that apply to your setup.