Free robots.txt Generator
Generate an AI-ready robots.txt for your website. Allow AI search engines and research agents to cite your brand while protecting private endpoints and controlling model training scrapers.
How to deploy your robots.txt in 4 steps
Choose Your Bot Policy
Select whether you want to allow AI search citation bots (like OAI-SearchBot and PerplexityBot) while optionally blocking model training scrapers (like GPTBot and CCBot).
Save to Your Web Server Root
Download the generated file and place it in your public root directory at /public/robots.txt so it resolves at yourdomain.com/robots.txt.
Verify Your Live File
Open yourdomain.com/robots.txt in your browser or test with a validator to confirm search bots can read your directives without SSL or permission errors.
Pair with an llms.txt File
While robots.txt gives crawlers access permissions, an llms.txt file gives them structured content to cite. Deploy both for maximum AI visibility.
Frequently Asked Questions
Track which AI crawlers are visiting your website
Vestigo monitors live AI crawler traffic, tracks brand citations across ChatGPT and Perplexity, and alerts you when new competitors outrank your content.