AI crawlers and WooCommerce: let GPTBot, PerplexityBot and Claude read your store (robots.txt + Cloudflare, 2026)

If an AI assistant cannot fetch your pages, nothing else you do for AI visibility matters. Yet blocking AI crawlers has become the default on much of the web — Cloudflare enables it for new zones, security plugins ship bot blocklists, and hosting WAFs challenge anything that isn’t a browser. For a store that wants to be recommended, this is self-inflicted invisibility.

Know the bots (and what each one does)

There are three kinds of AI-related agents. Confusing them is why so many stores block the wrong thing.

Vendor	Search / retrieval index (you want these)	User-triggered fetch (you want these)	Training crawler
OpenAI	OAI-SearchBot	ChatGPT-User	GPTBot
Perplexity	PerplexityBot	Perplexity-User	—
Anthropic	Claude-SearchBot	Claude-User	ClaudeBot
Google	Googlebot (also feeds AI Overviews / AI Mode)	Google-Agent and other user fetchers	Google-Extended (a robots token; blocking it does not remove you from AI Overviews or AI Mode)
Microsoft	Bingbot (powers Copilot)	—	controlled by noarchive/nocache meta, not by user agent
Mistral	MistralAI-Index	MistralAI-User	MistralAI-Training
Apple	Applebot	—	Applebot-Extended (token)

Search bots decide whether you can be cited. User-triggered fetchers load your page when someone asks the assistant about you right now. Training crawlers feed future models. A store should allow the first two without hesitation; allowing training crawlers is a business decision, but note that being in a model’s training data is how a brand ends up in its “memory”.

robots.txt for a store that wants AI visibility

The simplest correct file allows everything except private paths:

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Sitemap: https://yourstore.com/sitemap_index.xml

Explicitly listing OAI-SearchBot, PerplexityBot, Claude-SearchBot with Allow: / is harmless and makes intent visible to audits, but not required when * already allows.

Two traps: user-triggered fetchers (ChatGPT-User, Perplexity-User) mostly ignore robots.txt anyway — the real gate is your firewall. And for Bing/Copilot, avoid noarchive/nocache meta tags on pages you want cited.

Cloudflare: five switches to check

1. Security → Settings → AI bot policy. Since July 2025 new zones default to block. From 15 September 2026 the setting is split into Search / Agent / Training; set the ones you want to Allow — and note Cloudflare warns that blocking “Training” can also block multi-purpose crawlers such as Googlebot and Bingbot on affected pages.
2. AI Crawl Control. Set each AI crawler to Allow (not Block or Charge). Check the robots.txt-violation view.
3. Bot Fight Mode / Super Bot Fight Mode. Free-plan Bot Fight Mode challenges pattern-matched bots and cannot be skipped by WAF rules; either turn it off or upgrade and set Verified bots = Allow. If you use Cloudflare Tunnel, keep Definitely automated = Allow.
4. Managed robots.txt. This feature prepends Disallow: / rules for AI training bots and a Content-Signal line to your file. Turn it off, then confirm your live /robots.txt shows only your rules.
5. WAF custom rules and rate limits. Look for rules matching bot user agents or cloud-provider ASNs, and skip bot products for cf.verified_bot_category in AI Crawler / AI Search / AI Assistant.

Hosting and WordPress plugins

• Security plugins (Wordfence, All-in-One Security, etc.) often ship “bad bot” lists that include AI user agents. Whitelist the search and user-triggered agents above.
• Some managed WordPress hosts block bots at the edge; ask support for their AI-crawler policy.
• Caching layers must not serve HTML fragments or challenge pages to non-browser agents.

Verify from the outside

curl -sI -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://yourstore.com/
curl -sI -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourstore.com/

You want HTTP/2 200 and real HTML, not a 403, a challenge page or a redirect loop. Repeat for a product page and a category page. Then look at Cloudflare Security Analytics for blocked requests carrying those user agents.

What about llms.txt?

Google has said Search does not use it; OpenAI’s crawler documentation recommends robots.txt, not llms.txt; and Ahrefs found that 97% of llms.txt files across 137,000 sites were never requested. It is harmless to publish, but crawler access, feeds and schema are where the results come from.

Not sure whether your store is readable? Our free AI visibility check includes an outside-in crawler test for every major AI agent.

---

Sources: OpenAI bot documentation; Perplexity bot documentation; Anthropic crawler support article; Google Search Central — common crawlers and “AI features and your website”; Cloudflare — “Introducing Pay Per Crawl” (July 2025), “Content Independence Day” (July 2026), AI Crawl Control and Bot Fight Mode documentation; Ahrefs llms.txt study (June 2026).