Robots.txt Generator
Generate a valid robots.txt with your disallow rules, sitemap reference, and optional blocks for the major AI training and answer-engine crawlers. Whether to block those is a real strategic decision, not an obvious yes.
Free, no signup, and everything runs in your browser.
Being crawlable is step one.
InkSTR tracks whether ChatGPT, Perplexity, Gemini, and Claude are actually citing your pages, so you can see what allowing those crawlers is earning you.
Common questions
About robots.txt generator.
More free tools
No signup on any of them.
UTM Link Builder
Build tagged campaign URLs that report cleanly in analytics.
ExploreOccupancy, ADR and RevPAR Calculator
The three numbers that describe a rental's performance.
ExploreDirect Booking Savings Calculator
What shifting bookings off the OTAs is actually worth.
Explore
Stop doing this by hand every week.
InkSTR researches the keywords, writes the articles, and publishes them to your site on a schedule.
3 free articles · No credit card required
Turn this off only for a staging site you want fully blocked.
One per line. A leading slash is added if you leave it off.
Seconds between requests. Google ignores this; Bing and others honour it.
Adds explicit rules for 10 agents including GPTBot, ClaudeBot, and PerplexityBot.
Save as /robots.txt
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /*?s=
Sitemap: https://example.com/sitemap.xmlrobots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other sites link to it, so to genuinely keep a page out of the index you must allow crawling and use a noindex meta tag instead. The file has to sit at the root of the domain, exactly at yoursite.com/robots.txt, and each subdomain needs its own.