🤖

Robots.txt Generator

Generate robots.txt files with presets for blocking AI bots, search engines, and custom rules.

Quick Presets

User-Agent Rules

User-Agent

Paths

Sitemap URL (optional)

Crawl-delay (optional, in seconds)

Generated robots.txt

User-agent: *
Allow: /

About Robots.txt Generator

Free robots.txt generator with quick presets and full customization. Create a properly formatted robots.txt file for your website in seconds. Choose from presets like Allow All, Block All, Block AI Bots (GPTBot, ChatGPT-User, CCBot, Bytespider), or Standard recommended settings. Add custom rules for any user-agent with multiple Allow and Disallow paths. Include your sitemap URL and optional crawl-delay. See a live preview of your robots.txt file as you configure it, then copy to clipboard or download as a file. Essential for controlling how search engine crawlers and AI bots access your website content.

Frequently Asked Questions

What is a robots.txt file?▾

A robots.txt file is a text file placed at the root of your website (e.g., example.com/robots.txt) that tells search engine crawlers and bots which pages they can and cannot access. It follows the Robots Exclusion Protocol standard.

How do I block AI bots from crawling my site?▾

Add Disallow rules for AI bot user-agents: GPTBot (OpenAI), ChatGPT-User (ChatGPT browsing), CCBot (Common Crawl/Anthropic training data), and Bytespider (ByteDance). This tool has a one-click 'Block AI Bots' preset that sets this up for you.

Does robots.txt guarantee bots won't access my site?▾

No. robots.txt is a voluntary standard - well-behaved bots (Google, Bing, most AI crawlers) respect it, but malicious bots may ignore it. For sensitive content, use server-side authentication or access controls instead.

Where should I place my robots.txt file?▾

The robots.txt file must be placed at the root of your domain (e.g., example.com/robots.txt). It will not work in subdirectories. Search engines only look for it at the root URL. The file should be plain text with UTF-8 encoding.

Is this robots.txt generator free?▾

Yes, completely free with no signup. Choose from presets or create custom rules, add your sitemap URL, preview the output, and copy or download the file instantly. All processing happens in your browser.

Related Tools

🔎

Meta Description Generator

Generate SEO-optimized meta descriptions with a live Google SERP preview. Free, no signup.

🔍

Meta Tag Generator

Generate SEO-optimized meta tags for your website with live preview.

📊

Keyword Density Checker

Analyze keyword density and frequency in your content. Find top 1-word, 2-word, and 3-word phrases.

🔗

Open Graph Generator

Generate Open Graph and Twitter Card meta tags with live social media previews.

🗺️

Sitemap Generator

Generate a valid XML sitemap for your website - add URLs, set priorities and change frequencies, then download the sitemap.xml file.

📊

Schema Markup Generator (JSON-LD)

Generate JSON-LD structured data for Google rich results. Supports Article, FAQ, Product, Recipe, Event, and more.

Browse all 139+ free tools