Skip to content
aivoma
en
Free check

Fix guides · Crawler access

Your firewall answers browsers but rejects AI crawlers

Cloudflare "Block AI bots", Bot Fight Mode and WAF rules return 403 to GPTBot, OAI-SearchBot and PerplexityBot even when robots.txt allows them. Which switches to change.

Impact: Critical We fix this in AI Visibility Setup — €790 · Updated

The symptom

  1. robots.txt looks fine, yet AI assistants never cite your store.
  2. Requests with an AI-crawler user agent get a 403, 429 or a Cloudflare challenge page while normal browsers get 200.
  3. Cloudflare dashboard shows "AI Scrapers and Crawlers: Block" (default for zones created since July 2025) or Bot Fight Mode enabled.

Why it matters

robots.txt is only a request. The edge (Cloudflare, Sucuri, Wordfence, the host’s own WAF) enforces. When the edge returns 403, the crawler records the page as inaccessible and stops trying, so nothing on the store can be quoted.

Cloudflare turned on AI bot blocking by default for new zones in 2025, and from 15 September 2026 splits it into Search, Agent and Training categories. Many owners never opened that panel.

Bot Fight Mode and Super Bot Fight Mode challenge "definitely automated" traffic with JavaScript — which crawlers cannot solve.

How to check it yourself

  1. From any terminal: curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://yourstore.com/ — a 403/429/503 means the edge is blocking. Compare with a browser user agent.
  2. Cloudflare → Security → Bots: check "AI Scrapers and Crawlers" / "Block AI bots" and Bot Fight Mode. Security → AI Crawl Control lists each crawler with Allow/Block.
  3. Security → WAF → Custom rules: look for rules matching cf.verified_bot_category eq "AI Crawler" or user-agent strings.
  4. The free Aivoma check performs exactly this edge probe for OAI-SearchBot and GPTBot.

How to fix it

  1. Cloudflare → Security → Bots: set AI bot blocking to *Allow* (after September 2026: Search = Allow, Agent = Allow; Training is your choice). Turn Bot Fight Mode off; if you need Super Bot Fight Mode, set *Verified bots* to Allow.
  2. Security → AI Crawl Control: set OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Google-Extended and Bingbot to *Allow*.
  3. WAF custom rules: delete or add a skip rule for verified bots: expression cf.client.bot → action *Skip* (all remaining features).
  4. Managed robots.txt: turn off Cloudflare’s managed robots.txt, otherwise it re-adds AI disallow lines on top of yours.
  5. Wordfence / Sucuri / hosting WAF: disable "block AI bots" or "block fake Googlebot"-style rules for verified crawlers; whitelist the user agents above.
  6. Re-run the curl test until you get 200 for both bot user agents.

FAQ

Straight answers to the questions store owners ask

Isn’t blocking AI bots good for security?

Verified AI crawlers identify themselves and respect robots.txt; blocking them stops recommendations, not attacks. Keep WAF managed rules and rate limiting on — those target real abuse.

I don’t use Cloudflare. Who else blocks?

Sucuri, Wordfence "rate limiting" for bots, Imunify360 on cPanel hosts, and some Nginx configs shipped by managed WooCommerce hosts. The curl test finds the block; the host’s support can lift it.

Free instant store check

See what AI assistants can read on your store — in 60 seconds

Paste your store URL. We test AI-crawler access, product schema, prices, GTINs, feeds and email security live, and show the exact gaps. No sign-up, no obligation.

Check my store free