Shopify robots.txt and AI crawlers: who to let in
Short version: Shopify's default robots.txt doesn't block AI bots. Stores usually block them by accident: an old custom robots.txt.liquid, a bot protection app, or a firewall rule. Blocking training bots is a fair choice. Blocking the search and answer bots means AI assistants can't see you.
Check your store's bot access and readiness grade →
Three kinds of AI bot
"AI crawler" covers bots with very different jobs, and the difference matters more than the names.
- Search bots build the index an assistant searches when it answers. OpenAI's is
OAI-SearchBot, Anthropic's isClaude-SearchBot, Perplexity's isPerplexityBot. Block these and you drop out of AI search results. - User bots fetch a page because a person just asked about it:
ChatGPT-User,Claude-User,Perplexity-User. OpenAI notes robots.txt rules may not apply to these, since a person started the request. - Training bots collect pages that may train future models:
GPTBot,ClaudeBot.Google-ExtendedandApplebot-Extendedaren't separate crawlers, they're switches that say whether Google and Apple may use what their normal crawlers fetch for training.
A store that wants to sell through AI assistants should let the search and user bots in. Training bots are a judgement call. Blocking them doesn't remove you from answers today.
What Shopify does by default
Shopify generates robots.txt for every store. The default rules keep all bots out of the admin, cart, checkout, account pages and search result URLs, and name a few SEO crawlers. They don't single out AI bots, so out of the box GPTBot and friends follow the same rules as Googlebot.
How stores end up blocking AI bots
- A custom robots.txt.liquid. Someone added
Disallow: /for GPTBot in 2023, when blocking AI was the fashionable move, and nobody revisited it. - A bot protection or anti-scraping app that treats anything not a browser as hostile.
- A firewall in front of the store (for headless stores, a CDN's "block AI bots" setting).
How to check and fix it
- Open
yourstore.com/robots.txtand look forUser-agentlines naming AI bots, followed byDisallow: /. Or run our agent readiness check, which also knocks on your front door as each bot to catch firewall blocks. - To edit rules, go to Online Store → Themes → Edit code on your live theme and open (or add) the
robots.txt.liquidtemplate. Keep Shopify's default block and add your own rules after it, so you don't lose the sensible defaults. - If robots.txt looks fine but bots are still turned away, check bot protection apps and any firewall or CDN settings.
A minimal rule set that welcomes AI search and answer bots but opts out of training looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Everything not named there, including OAI-SearchBot, Claude-SearchBot and PerplexityBot, falls back to Shopify's default rules.
Questions
Does blocking GPTBot keep my store out of ChatGPT?
Not by itself. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when someone asks. Products in ChatGPT shopping also come through Shopify's own feeds.
Will allowing AI bots slow my store down?
The major AI bots crawl politely and Shopify serves storefront pages from its CDN, so most stores won't notice. If one bot misbehaves, block that one rather than all of them.
Where is robots.txt in Shopify?
Shopify generates it. To change it, add or edit the robots.txt.liquid template under Online Store, Themes, Edit code.