Free AI Bot Access Checker
Scan your live website against 23 AI crawlers. Check which search engines, research agents, and model training bots can discover and cite your content.
How to configure AI bot permissions
Distinguish Search Bots from Training Crawlers
Search bots (like OAI-SearchBot and PerplexityBot) retrieve live answers for users. Training crawlers (like GPTBot and CCBot) scrape datasets for future models.
Allow Citation Bots for Brand Visibility
Ensure citation crawlers can access your public documentation, product pages, and blog posts so AI engines can quote and recommend your business.
Protect Private Endpoints
Add Disallow rules for internal application paths such as /api/, /dashboard/, /admin/, and /auth/ so bots do not waste crawl budget on authenticated routes.
Combine with an llms.txt File
Granting crawler access is step one. Providing a clean /llms.txt markdown file helps AI models digest your product offerings without parsing complex HTML.
Frequently Asked Questions
Track AI search citations automatically
Vestigo continuously monitors ChatGPT, Perplexity, Claude, and Google AI Overviews to show when your brand is cited and which competitors get recommended.