Skip to content
aivoma
en
Free check

Fix guides · Crawler access

Your robots.txt is telling ChatGPT and Perplexity to stay out

If robots.txt disallows GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot or Google-Extended, AI assistants cannot read or recommend your WooCommerce store. Exact rules to fix it.

Impact: Critical We fix this in AI Visibility Setup — €790 · Updated

The symptom

  1. Your store never appears in ChatGPT, Perplexity or Copilot answers, even for your own brand name.
  2. https://yourstore.com/robots.txt contains lines such as User-agent: GPTBot followed by Disallow: /, or a blanket User-agent: * + Disallow: /.
  3. A security, SEO or "AI blocker" plugin was installed in 2024–2025 and quietly added the rules.

Why it matters

AI assistants use dedicated crawlers to read and cite web pages: OAI-SearchBot (ChatGPT search answers), GPTBot (OpenAI training and some retrieval), PerplexityBot, ClaudeBot, Google-Extended (Gemini grounding) and Bingbot (Copilot). If robots.txt disallows them, they do not fetch your pages — and a store that cannot be read cannot be recommended.

Many WordPress "block AI" plugins target all of these bots at once. That may be reasonable for a news site worried about training data; for a shop it removes you from the fastest-growing source of high-intent buyers.

Blocking OAI-SearchBot specifically is the single most damaging rule: it is the bot that powers ChatGPT shopping and search answers, not training.

How to check it yourself

  1. Open https://yourstore.com/robots.txt in a browser. Look for groups naming GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended, CCBot or Bingbot with Disallow: /.
  2. Check the catch-all group: User-agent: * with Disallow: / blocks everyone, including Google.
  3. Run the free Aivoma check below — it parses robots.txt for every AI crawler and also tests whether the server accepts them at the edge.

How to fix it

  1. Find who writes the file. WordPress generates robots.txt virtually; Rank Math (SEO → General → Edit robots.txt), Yoast (Tools → File editor), All in One SEO and several security plugins can override it. If a physical robots.txt exists in the web root, it wins over all plugins.
  2. Remove the AI-bot disallow rules, or replace them with explicit allows. A safe baseline for a WooCommerce store:
  3. User-agent: * · Disallow: /wp-admin/ · Disallow: /cart/ · Disallow: /checkout/ · Disallow: /my-account/ · Disallow: /*?add-to-cart= · Allow: /wp-admin/admin-ajax.php · Sitemap: https://yourstore.com/sitemap_index.xml
  4. Do not disallow /?s=, /product/, /product-category/ or your images directory; assistants and Google Shopping need product pages and images.
  5. If you want to keep training crawlers out while staying visible, allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Bingbot, and disallow only GPTBot, CCBot and Google-Extended. Note that Google-Extended controls Gemini grounding, so blocking it can cost Gemini/AI Mode mentions.
  6. After saving, re-open robots.txt in a private window (caches!) and re-run the check. Changes are picked up by crawlers within days.

FAQ

Straight answers to the questions store owners ask

Does blocking GPTBot stop ChatGPT from recommending my store?

Partly. ChatGPT search answers rely mainly on OAI-SearchBot, so blocking only GPTBot usually leaves search answers working. Blocking OAI-SearchBot or ChatGPT-User removes you from answers and from link previews. Most plugins block all three at once.

Will allowing AI crawlers slow down my store?

Not noticeably. AI crawlers fetch far less than Googlebot, and the checked site should already cache pages. If you see real load, rate-limit them in Cloudflare rather than blocking.

I removed the lines but the file still shows them.

Clear your page cache (LiteSpeed, WP Rocket, W3TC) and any Cloudflare cache, and check whether a physical robots.txt file in the web root overrides the plugin.

Free instant store check

See what AI assistants can read on your store — in 60 seconds

Paste your store URL. We test AI-crawler access, product schema, prices, GTINs, feeds and email security live, and show the exact gaps. No sign-up, no obligation.

Check my store free