← Tools
AI crawler access check
Reads your robots.txt and shows, in a table, which of 16 AI crawlers may enter your site.
The audit only reads publicly available files. It writes nothing and signs in to nothing.
Short answer
Can GPTBot or ClaudeBot crawl my site?
The AI crawler access check fetches a domain's /robots.txt, interprets it under RFC 9309, and returns allowed, blocked or unspecified for 16 crawlers including GPTBot, ClaudeBot, PerplexityBot and Google-Extended. It applies group matching and the longest-match rule, which is not the same thing as searching the file for 'Disallow'.
Key takeaways
- If a crawler has its own group, the wildcard (*) group does not apply to it at all — not even for rules its own group leaves out.
- When Allow and Disallow collide, the longest pattern wins, not the line that comes first.
- 'Unspecified' is not the same as 'allowed': the crawler may enter today, but the site has made no statement about it.
Frequently asked
Should I block AI crawlers?
- These are two separate decisions and should not be confused. Blocking training crawlers (GPTBot, Google-Extended, ClaudeBot) limits your content being used to train models, and is a legitimate licensing choice. Blocking the crawlers that produce answers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) removes your brand from those answers entirely.
Queries this page answers
- gptbot robots.txt
- block ai crawlers
- is gptbot blocked
- llms.txt validator
Let's talk about your project.
A new brand, a website that needs rebuilding, or visibility in search — tell us where you want to start and we will map the route with you.