A web developer working on code in a modern office setting with multiple devices.

Cloudflare’s AI crawler controls for WordPress: what changed, what conflicts, and how to use them without breaking your site

BySplinternet Marketing

Cloudflare’s AI crawler tooling has matured. You now have three distinct controls: AI Crawl Control, Block AI Bots, and AI Labyrinth.

The problem I’m seeing on WordPress sites isn’t a lack of features. It’s overlap and rule conflicts. Cloudflare explicitly documents that WAF custom rules and other bot or security settings can affect how AI Crawl Control behaves. If you already run Bot Fight Mode, Super Bot Fight Mode, custom WAF expressions, or rate limits, your new AI settings may never execute the way you expect.

Before you flip a switch, you need to understand what each control actually does — and where precedence matters.

What each Cloudflare AI crawler control actually does

AI Crawl Control (as documented in Cloudflare’s developer docs) gives you visibility into identified AI crawlers and lets you choose per-crawler actions. Depending on plan and detection visibility, you can monitor, allow, or block specific AI crawlers. It’s designed for policy control — not blanket bot mitigation.

Block AI Bots is a broader setting under Cloudflare’s bot configurations. It’s a higher-level control meant to block known AI bots more generally, rather than managing them individually. This is appropriate when your policy is simple: no AI crawler access.

AI Labyrinth, introduced by Cloudflare and described in both the developer documentation and announcement post, targets crawlers that ignore directives. Instead of relying on robots.txt compliance, it serves deceptive link structures to waste scraper resources and reduce origin impact. Cloudflare positions it as designed to avoid visual and SEO impact while targeting automated scraping behavior.

These are not interchangeable:

  • AI Crawl Control = visibility and per-crawler policy.
  • Block AI Bots = broader blocking of known AI bots.
  • AI Labyrinth = mitigation for non-compliant crawlers that keep hitting origin resources.

Also note: Cloudflare’s documentation makes clear that crawler identification and feature availability can vary by plan. If you don’t see granular visibility, it may not be a misconfiguration — it may be plan-based detection limits.

Where WordPress operators get into trouble

Cloudflare documents that WAF custom rules can affect AI Crawl Control outcomes. In practice, that means:

  • A custom WAF rule blocking a user agent pattern may execute before your AI Crawl Control setting.
  • Bot Fight Mode or Super Bot Fight Mode may challenge or block traffic before your per-crawler rule applies.
  • Rate limiting rules may throttle traffic you intended to “monitor.”

If you’ve hardened your WordPress site over time — especially after scraping incidents — you likely have layered controls. That’s good security hygiene. But layering without auditing precedence creates false assumptions.

I’ve seen teams enable AI Crawl Control, set a crawler to “monitor,” and assume they’re allowing it — while an older WAF expression is still blocking that traffic upstream. The dashboard suggests one thing. The edge is doing another.

Another common failure mode: enabling Block AI Bots while also configuring per-crawler allowances in AI Crawl Control. If the broader block executes first, your granular rule may never matter.

For WordPress and WooCommerce sites, the business impact isn’t abstract:

  • Unnecessary origin load from scrapers that weren’t actually mitigated.
  • Blocked traffic you thought you were allowing.
  • Duplicated defenses that complicate troubleshooting.
  • Time wasted debugging analytics anomalies that are really edge-rule conflicts.

What to do next

1. Inventory existing Cloudflare controls.
Review WAF custom rules, Bot Fight Mode, Super Bot Fight Mode, rate limits, and any managed bot protections. Document what executes and in what order.

2. Decide your policy before touching settings.
Do you want:

  • Visibility first? Start with AI Crawl Control in monitor mode.
  • No AI access at all? Consider Block AI Bots as a broad control.
  • Granular policy by crawler? Use AI Crawl Control and avoid overlapping blanket blocks.
  • Mitigation for non-compliant scrapers? Add AI Labyrinth where repeated abuse persists.

3. Remove duplicate logic.
If a custom WAF rule already blocks specific AI user agents, decide whether to retire that rule in favor of AI Crawl Control — or vice versa. Avoid running both unless you understand precedence.

4. Validate in logs and analytics.
Check Cloudflare security events, bot analytics, and origin server logs (via cPanel/WHM if applicable). Confirm that the traffic you intended to block is actually blocked — and that allowed traffic reaches origin as expected.

5. Measure origin impact.
Watch origin CPU, PHP worker usage, and request volume. The goal isn’t just policy clarity — it’s lower wasted requests, reduced scraping load, and simpler operations.

Robots.txt alone does not enforce anything. Compliant crawlers may honor it; hostile ones may not. Cloudflare’s newer AI controls give you more leverage — but only if you understand how they interact with the security stack you already built.

A short audit this week can prevent months of incorrect assumptions about what your edge is actually enforcing — and that translates directly into lower maintenance burden and more predictable performance for your WordPress site.

Sources

Know someone who would benefit from this update? Share this article with them.

This article is for informational purposes only and reflects general marketing, technology, website, and small-business guidance. Platform features, policies, search behavior, pricing, and security conditions can change. Verify current requirements with the relevant platform, provider, or professional advisor before acting. Nothing in this article should be treated as legal, tax, financial, cybersecurity, or other professional advice.

Related Posts

Splinternet Marketing