Plenty of sites serve browsers fine but return 403 to AI crawlers. Here's how to check GPTBot, ClaudeBot and PerplexityBot access yourself with one curl line.
When we measured ten K-beauty exporters on one day, three were returning 403 to ChatGPT's crawler (GPTBot). They loaded fine in a browser but shut the door on AI — and none of the three knew. The usual cause is a misfiring CDN security rule. The good news: checking takes five minutes and fixing takes a few config lines.
If GPTBot is blocked, ChatGPT cannot pull a single fact from that brand's official site. When an overseas shopper asks "what's this brand like?", the AI builds its answer only from third parties — retailer pages, old blogs, forums. The brand loses control of its own story. This is exactly what our "technical access" dimension measures.
Not comfortable with a terminal? Stop here. What follows is the developer/marketer route. If the commands feel out of reach, request a free scorecard and we'll check your site's crawler access for you and report back within two days — you can just receive the result.
If you do use a terminal, it's two lines. First, pretend to be a normal browser:
Then pretend to be GPTBot and compare:
If the first line is 200 but the second is 403 or 429, you're blocked. Repeat with the other user-agents below — just swap the UA token.
| Crawler | User-agent token | For |
|---|---|---|
| GPTBot | GPTBot | ChatGPT training/search |
| OAI-SearchBot | OAI-SearchBot | ChatGPT search citations |
| ClaudeBot | ClaudeBot | Claude |
| PerplexityBot | PerplexityBot | Perplexity |
Sometimes the block is an explicit robots.txt rule rather than the CDN. Open https://yourbrand.com/robots.txt; if you see Disallow: / under User-agent: GPTBot, that's the cause — deleting a few lines fixes it.
On a CDN (e.g. Cloudflare), add the four AI crawlers to the allow list in your bot rules. For a robots.txt issue, remove the Disallow line. Both are instant fixes, which is why our reports always rank them first. Re-run the curl above and confirm a 200. This is the same method we used for the K-Beauty Index.
The free scorecard runs these checks against your live pages and tells you which of the three pillars is your bottleneck — in two business days, no charge.