Check your robots.txt

See which search engines and AI crawlers your robots.txt lets in and keeps out. No file yet? Make one in a minute.

What do you want to do?

Free. No account needed.

Your robots.txt
  • Search enginesGoogle and BingWaiting
  • AI assistantsChatGPT, Claude, Perplexity and GeminiWaiting
  • AI trainingOpenAI and AnthropicWaiting

Your result shows here

What robots.txt does

robots.txt is a plain text file at the top of your site. Search engines and AI crawlers read it before anything else to see which pages they may open. Each group of lines names a crawler, then lists what it may and may not read. A Sitemap line tells them where your list of pages is.

What this tool checks

We read your file and ask it one question for each crawler we know: may it open your home page? That covers Google and Bing, the crawlers ChatGPT, Claude, Perplexity and Gemini use to answer people, and the ones OpenAI and Anthropic use to train new models. We also list the sitemaps your file points to.

What it doesn't look at

A firewall can turn crawlers away even when robots.txt lets them in, and a rule for one folder doesn't show up in a check of your home page. For one page and the exact fix, use the AI crawler check.

Get read. Then get recommended.

AutoSEOPilot writes the answers your buyers ask for and shows you when ChatGPT and Google start naming you.

Start free trial

Questions people ask.

Is this tool free?
Yes. Check a site, paste a file or make a new one. No account needed.
Where does robots.txt go?
In the top folder of your site, so it opens at yourwebsite.com/robots.txt. Most site builders and SEO plugins have a setting for it.
What if I don't have one?
Then crawlers are let in by default. A file still helps: it points them to your sitemap and can keep private pages out.
Should I keep AI crawlers out?
If you don't want AI assistants to read your pages, yes. Keeping them out means ChatGPT, Claude, Perplexity and Gemini can't use your pages when people ask them for help.
What does opting out of AI training do?
It asks OpenAI and Anthropic not to train future models on your pages. Their assistants use separate crawlers for answers, so they can still show you.
Does robots.txt hide a page from Google?
It stops Google reading the page, but a page can still show up in results if other sites link to it. To keep a page out of results, use a noindex tag instead.