What is robots.txt and how do you write one?

What is robots.txt and where does it live?

robots.txt is a plain-text file telling crawlers which parts of a site they may fetch. It counts only at the root: for example.com the file read is /robots.txt, and example.com/blog/robots.txt means nothing. Every subdomain owns its own file — blog.example.com needs a separate one, and the rules on the main domain do not carry across.

The file became a formal standard as RFC 9309 in 2022 — unlike llms.txt, it is an agreement crawlers have jointly accepted. It is still not a security control: honouring the rules is up to the crawler, and a malicious scraper walks straight past without reading the file. Writing a path that must stay private into robots.txt is the same as announcing it on a public list.

How do you write a robots.txt file?

The file is made of groups. Each group opens with one or more User-agent lines, followed by the rules that apply to that crawler. Four directives are enough:

• User-agent: names the crawler the rules are written for. The wildcard (*) means 'everyone without their own group' — not 'everyone'.
• Disallow: blocks fetching of addresses starting with the given path. Left empty, it blocks nothing.
• Allow: reopens a path from inside a Disallow; its real use is one open folder beneath a closed directory.
• Sitemap: the full address of the sitemap. It is independent of groups, may sit anywhere in the file, and is conventionally written last.

# example.com/robots.txt

# Default: every crawler without a group of its own.
User-agent: *
Disallow: /cart/
Disallow: /account/
Disallow: /search
Disallow: /*?sort=
Allow: /

# Training crawlers — a content licensing decision.
# Deleting the two lines below closes your content to training.
User-agent: GPTBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Allow: /

# Search and live fetch — this is where visibility comes from.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Sitemap: https://example.com/sitemap.xml

Which rule wins? Group matching and longest match

This is the most misunderstood part of the file, and it comes down to two rules. First, group selection: a crawler finds the group whose user agent matches it most specifically and applies only that one. Google's own documentation is explicit — a specific group and the wildcard group are never combined.

User-agent: *
Disallow: /private/
Disallow: /admin/

User-agent: GPTBot
Disallow: /training-data/

# Intent: keep GPTBot out of everything, and keep /training-data/ closed to all.
# Reality: because GPTBot has its own group, the two lines above never apply to it.
#          GPTBot can now fetch /private/ and /admin/ freely.

The second rule governs conflicts inside one group: the longest path pattern wins. Where the lengths are equal, Google applies the least restrictive rule — Allow. That is why 'Disallow: /' together with 'Allow: /blog/' leaves the blog open: the Allow pattern is longer.

The practical consequence: searching a robots.txt for 'Disallow' proves nothing. Which rule applies is decided by group matching and pattern length, and the conclusion you reach by reading the file with your eyes is usually wrong.

Which AI crawlers exist, and what does each do?

AI crawlers do three different jobs, and the decision should split three ways. Training crawlers collect content for the dataset that ends up in a model's weights. Search crawlers index the sources shown when a user asks a question — brand visibility depends directly on these. Live-fetch crawlers pull a page at the moment a user opens a specific link.

OpenAI's own documentation notes that for ChatGPT-User 'robots.txt rules may not apply', on the grounds that the request is a user's own action rather than automated crawling. If you plan to block that bot, plan knowing the outcome is not guaranteed.

Which bots should you block, and which should stay open?

The decision follows the business model, and there is no single right answer. For publishers whose content is the product — news sites, research publications, paid archives — blocking training crawlers is defensible: the content itself is what is sold. For companies selling services and products the arithmetic reverses: content is not the thing sold but the thing that leads to the sale.

In either column, blocking search and live-fetch crawlers is rarely the right call. It is the generative-engine equivalent of refusing to be findable in search — and the cost does not show up immediately, it shows up months later as competitors standing alone in the answers.

When does a robots.txt change take effect?

Not immediately — and not knowing that causes a lot of unnecessary panic. Google generally caches robots.txt for up to 24 hours, and longer where refreshing is not possible. OpenAI's documentation states that it takes roughly 24 hours from a robots.txt update for its systems to adjust.

The file being unreachable is also a decision. When robots.txt returns 5xx, Google stops crawling for 12 hours while retrying; if fetching keeps failing it uses the cached version for up to 30 days. Once that runs out, and if the site is generally available, it behaves as though there were no restrictions. A server error, past a certain point, means 'everything is allowed'.

One last limit: Google enforces a robots.txt size limit of 500 kibibytes and ignores content beyond it. Hand-written files rarely get near it, but large e-commerce sites with auto-generated Disallow lists do — and the rules past the cut are ignored silently.

I allowed it in robots.txt but the bot still cannot get in: the firewall trap

The most common reason a crawler cannot reach a site is not robots.txt. robots.txt is a request; the CDN and firewall layer never lets the request reach the server at all. Marketing opens the file, a switch on the infrastructure side drops the bot at the edge, and for months neither side knows where the problem is.

Cloudflare is the most widespread example. The 'Block AI bots' switch in security settings blocks verified bots classified as crawling for AI training, plus a number of unverified bots that behave similarly; per the documentation it excludes mixed-purpose bots used for both training and search. Cloudflare is deprecating that single switch on 15 September 2026, replacing it with policies that let you manage search, agent and training behaviours separately.

• In Cloudflare, check the 'Block AI bots' switch under Security Settings and Bot Fight Mode; both operate independently of robots.txt.
• Look through WAF managed rule sets for blocking rules written against bot categories — they were usually switched on years ago and forgotten.
• On the server, check whether ModSecurity, fail2ban or rate-limiting rules are dropping crawler user agents.
• If a 'bad bot' or security plugin is installed — common on WordPress — open and read the list of user agents it blocks.
• If geo-blocking is in place: crawler egress IPs sit outside Turkey, and country-level blocking cuts them off too.

How do I verify what I have blocked or opened?

Three separate checks are needed, and each answers a different question: what does the file say, what is the edge layer doing, and did the bot actually arrive.

# 1) The file itself
$ curl -s https://example.com/robots.txt | head -40

# 2) Request a real page using the bot's user agent.
$ curl -s -o /dev/null -w "%{http_code}\n" \
    -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
    https://example.com/

# 200 -> it gets through
# 403 / 503 / 429 -> not robots.txt; the edge layer is blocking it

# 3) Confirm in the server log that the bot actually arrived.
$ grep -Ei "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log \
    | awk '{print $1, $12, $13}' | sort | uniq -c | sort -rn | head

What is the difference between robots.txt and noindex?

robots.txt blocks crawling, noindex blocks indexing — and confusing the two produces one of the most insidious SEO mistakes there is. Block a page in robots.txt and the crawler cannot fetch it; because it cannot fetch it, it never reads the noindex tag on the page. If the page is linked from elsewhere it can still be listed by URL alone, and the route to removing it has just been closed.

The right order is this: if you want a page out of the index, leave it open in robots.txt and put noindex on the page. The crawler enters, reads the tag, drops the page. Once it is out, you can close crawling in robots.txt as well if you want to.

What are the most common robots.txt mistakes?

• Carrying the staging environment's 'Disallow: /' line into production. One line, the whole site. Every team without a robots.txt check on its release list lives through this once.
• Opening a crawler-specific group and forgetting to repeat the general restrictions inside it — the file then leaves the bot you meant to restrict freer than all the others.
• Blocking CSS and JavaScript directories. A crawler that cannot render sees partial content; this still happens on sites that close /wp-includes/ or /assets/.
• Writing a full address in a Disallow line. Rules start with a path (/sepet/), not a domain; a line reading https://example.com/sepet/ does nothing at all.
• Writing the Sitemap line as a relative path. The sitemap requires an absolute address: not /sitemap.xml but https://example.com/sitemap.xml.
• Missing case sensitivity. Paths are case-sensitive; /Urunler and /urunler are two different paths.