Skip to content

Free tool · no sign-up for your score

Robots.txt AI bot checker

The SiteSignal robots.txt AI bot checker is a free tool that reads your robots.txt and reports, bot by bot, whether known AI crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot are allowed, blocked or unclear.

Website readiness only: we do not query ChatGPT, Perplexity or Google live, and we do not estimate lost customers. About 30 seconds as an estimate; some sites take up to 90.

Same engine, focused view

Every SiteSignal tool runs the full scan. This one opens your results on the part you came for, and keeps that focus after you unlock the full report.

See a sample report

Why robots.txt is where AI visibility starts

robots.txt is a small public file that tells automated bots what they may read. Many site templates and security plugins add blanket blocks without the owner knowing. One line can stop an AI product from learning anything from your site.

Bots that do different jobs

Some bots gather pages to train models. Others fetch pages live when a user asks a question. Google-Extended controls certain Google AI uses and does not affect normal Google Search. Treat them separately.

  • GPTBot: OpenAI's crawler
  • ClaudeBot: Anthropic's crawler
  • Google-Extended: a control token for some Google AI features
  • CCBot: Common Crawl, a public dataset many models use

Questions owners ask

  • Not always. It is a business choice. The problem is blocking them by accident. The full report shows each rule so you can decide on purpose.

  • Usually not. Without the file, bots assume they may read your public pages. We report it as unclear rather than failed.