Free Robots.txt Generator (AI-search friendly)

Generate a clean robots.txt that welcomes Google, Bing and AI crawlers while protecting private paths. Free, instant, no account needed.

Sitemap URL

Paths to keep private — one per line

Allow AI search crawlers (ChatGPT, Claude, Perplexity) — recommended for visibility Block pure training scrapers (CCBot, Bytespider)

Output

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Stop fixing by hand

SEOApex fixes this — and writes a ranking article every day.

Daily SEO + AI-search articles, hosted blog, Search Console insights and AI prompt tracking. 14-day free trial, cancel anytime.

Start free trial

More free tools

AI Crawlability Checker

See whether GPTBot, ClaudeBot, PerplexityBot and Google-Extended are allowed in your robots.txt, and whether you publish an llms.txt.

Robots.txt Checker

Fetch and validate your robots.txt: blocked search bots, disallow-all mistakes, sitemap declarations and AI crawler rules.

Sitemap Checker

Find your sitemap, confirm it loads, count the URLs inside and check it is declared in robots.txt.

Meta Title & Description Checker

Check title and meta description length, duplicates and missing tags on any page in seconds.

H1 Tag Checker

Detect missing, duplicate or multiple H1 tags and review your H2 outline for topical clarity.

Open Graph Checker

Preview how your page looks when shared on LinkedIn, X, Slack and WhatsApp — and fix missing OG tags.