Robots.txt Generator
Generate a valid robots.txt with your disallow rules, sitemap reference, and optional blocks for the major AI training and answer-engine crawlers. Whether to block those is a real strategic decision, not an obvious yes.
Free, no signup, and everything runs in your browser.
Being crawlable is step one.
InkSTR tracks whether ChatGPT, Perplexity, Gemini, and Claude are actually citing your pages, so you can see what allowing those crawlers is earning you.
Common questions
About the Robots.txt Generator.
It depends on what you want. Blocking GPTBot and ClaudeBot stops your content being used for training, but it also reduces the chance those assistants cite you when someone asks a question your page answers. For most businesses trying to be discovered, blocking them costs more than it protects.
No, and this is the most common misunderstanding. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To genuinely keep a page out of the index, allow crawling and use a noindex meta tag.
At the root of your domain, exactly at yoursite.com/robots.txt. It will not work in a subdirectory, and each subdomain needs its own file.
It is not required if you have submitted the sitemap in Search Console, but it is good practice. It helps crawlers that have not been told about it directly, including several AI crawlers that do not have a webmaster console.
More free tools
No signup on any of them.
Stop doing this by hand every week.
InkSTR researches the keywords, writes the articles, and publishes them to your site on a schedule.
3-day free trial · your first 3 articles included · nothing charged during the trial
Turn this off only for a staging site you want fully blocked.
One per line. A leading slash is added if you leave it off.
Seconds between requests. Google ignores this; Bing and others honour it.
Adds explicit rules for 10 agents including GPTBot, ClaudeBot, and PerplexityBot.
Save as /robots.txt
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /*?s=
Sitemap: https://example.com/sitemap.xmlrobots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other sites link to it, so to genuinely keep a page out of the index you must allow crawling and use a noindex meta tag instead. The file has to sit at the root of the domain, exactly at yoursite.com/robots.txt, and each subdomain needs its own.