Skip to content
aivoma
en
Free check

Fix guides · Crawler access

Your robots.txt blocks every crawler

A blanket Disallow: / in robots.txt — usually left over from staging or "Discourage search engines" — keeps a WooCommerce store out of Google, Shopping and every AI answer.

Impact: Critical We fix this in AI Visibility Setup — €790 · Updated

The symptom

  1. Google Search Console reports "Blocked by robots.txt" for most pages; organic traffic is near zero.
  2. robots.txt shows User-agent: * + Disallow: /.
  3. The store was recently migrated, cloned from staging or launched from "coming soon".

Why it matters

A blanket disallow stops Googlebot, Bingbot and every AI crawler at once. Nothing gets indexed, so nothing can be shown in Shopping, AI Overviews, ChatGPT or Perplexity.

WordPress writes this rule automatically when *Settings → Reading → Discourage search engines* is ticked — common on staging copies that then go live.

How to check it yourself

  1. Open /robots.txt. If the only rules are User-agent: * and Disallow: /, the store is closed.
  2. WordPress → Settings → Reading: is "Discourage search engines from indexing this site" ticked?
  3. Search Console → Pages → "Blocked by robots.txt".

How to fix it

  1. Untick *Discourage search engines* in Settings → Reading and save.
  2. If a physical robots.txt exists in the web root, edit it (or delete it so WordPress/your SEO plugin generates a sane one).
  3. Replace with the WooCommerce baseline shown in the GPTBot guide and add your sitemap URL.
  4. In Search Console, use *Validate fix* on the Blocked by robots.txt report; request indexing of key category pages.

FAQ

Straight answers to the questions store owners ask

How long until Google re-indexes?

Days for the home page and main categories, weeks for deep products. Submitting the sitemap in Search Console speeds it up.

Free instant store check

See what AI assistants can read on your store — in 60 seconds

Paste your store URL. We test AI-crawler access, product schema, prices, GTINs, feeds and email security live, and show the exact gaps. No sign-up, no obligation.

Check my store free