Decide who crawls what, including AI.
Start from a preset, add your sitemap and the paths to keep crawlers out of, then choose which AI crawlers may read your site. Copy or download a valid robots.txt.
AI crawlers
Your robots.txt
Upload the file to the root of your domain so it lives at /robots.txt. On WordPress, SEO plugins can edit a virtual robots.txt for you. Robots.txt controls crawling, not indexing: to keep a page out of Google, use a noindex tag and leave the page crawlable.
A robots.txt for how
search works now
Robots.txt used to be about Googlebot. Today the same file decides whether AI companies can train on your content and whether AI search engines can cite you. This handles both.
WordPress preset
Blocks /wp-admin/ while keeping admin-ajax.php open, the same default WordPress itself uses, so themes and plugins keep working.
Sitemap line
Add one or more sitemap URLs so every crawler that reads robots.txt can find your full list of pages.
Blocked paths
Keep crawlers out of carts, checkouts, internal search and other pages that waste crawl time.
AI crawler controls
Allow or block nine AI crawlers individually, from OpenAI, Anthropic, Google, Apple, Perplexity and Common Crawl.
Training vs search
One click blocks AI training bots while keeping AI search bots, so you can still be cited in ChatGPT and Perplexity answers.
Download ready
Copy the rules or download a robots.txt file to upload to the root of your domain.
How to use the robots.txt generator
- Choose a preset. Most sites should use WordPress or Allow everything.
- Add your sitemap URL and any paths crawlers do not need, such as cart or internal search pages.
- Decide on AI crawlers: allow all, block training only, or block all.
- Copy or download the file and place it at
yourdomain.com/robots.txt.
What robots.txt does, and what it does not
Robots.txt tells crawlers which URLs they may request. Reputable crawlers, including Googlebot and the major AI crawlers, follow it. It is a request, not a lock: it does not protect private content, and badly behaved bots can ignore it.
Crawling is not indexing
This is the most common mistake. Blocking a page in robots.txt stops Google from crawling it, but the URL can still be indexed if other pages link to it. To keep a page out of search, allow crawling and add a noindex meta tag. The meta tag generator can create one.
Never block your CSS and JavaScript
Google renders pages like a browser. Blocking theme files or scripts can stop it from seeing your layout and content properly.
Should you block AI crawlers?
It depends on what you want from AI. Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect content to train or ground AI models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so AI search products can show and link to them.
If you want visibility in AI answers, keep the search crawlers allowed. Blocking training bots is a separate choice about how your content is reused. Note that Google-Extended does not affect Google Search or AI Overviews, which use regular Googlebot.
Also check your sitemap is in shape: the XML sitemap generator builds one from a list of URLs.
Frequently asked questions
Where do I put my robots.txt file?
In the root of your domain, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file. On WordPress, SEO plugins such as All in One SEO, Yoast and Rank Math can edit it for you.
Does robots.txt stop a page from appearing in Google?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. Use a noindex meta tag on a crawlable page to keep it out of search.
Will blocking GPTBot remove my site from ChatGPT?
Blocking GPTBot stops OpenAI from using your pages for model training. ChatGPT search uses OAI-SearchBot, so keep that allowed if you want your pages to be shown and cited in ChatGPT search answers.
Does Google respect crawl-delay?
No. Googlebot ignores the crawl-delay rule. Some other search engines honour it, which is why it is optional here.
More free SEO tools
Meta Tag Generator
Title, description, canonical, robots and social tags as one paste-ready block.
Schema Markup Generator
JSON-LD for Article, FAQ, Organization, LocalBusiness and Product.
Hreflang Tag Generator
Reciprocal hreflang tags with x-default for every language version.