Robots.txt and
Sitemap Checker.
See exactly which crawlers can reach which pages, then count every URL your sitemaps hand to Google.
Your crawl rules and sitemaps appear here
See who is allowed in, test any path, and count every URL in your sitemaps.
How to use it
Three steps.
No account needed.
Enter your website
Type your domain. We read /robots.txt, then find and open every sitemap it lists, including sitemap indexes.
Test a path
Pick a crawler like Googlebot or GPTBot, type a path, and see instantly whether it is allowed and which line decides it.
Fix what is flagged
Each issue explains the fix in plain English. Download the full URL list from your sitemaps to compare with what you expect.
Questions
About the Robots.txt and Sitemap Checker
What is robots.txt for?
It is a plain text file at the root of your site that tells crawlers which paths they may visit. It controls crawling, not indexing: a blocked page can still show up in Google if other sites link to it. To keep a page out of results, use a noindex tag and leave it crawlable.
How does the path tester decide allowed or blocked?
It follows Google's published rules. A crawler obeys the most specific group that names it, or the * group if none does. Inside that group the longest matching rule wins, and when an Allow and a Disallow match equally, Allow wins. Wildcards (*) and end anchors ($) are supported.
Should I block AI crawlers like GPTBot?
That is a business decision. Blocking GPTBot or Google-Extended stops future model training on your pages, but blocking search focused crawlers such as OAI-SearchBot can keep you out of AI search answers. Use the tester to confirm your file does what you intend.
Why does my sitemap count look different from Search Console?
We count every URL listed in the sitemap files we can open, up to 15 files and 2 MB per file. Search Console shows how many of those Google chose to index, which is usually lower. Big gaps often point to duplicates, redirects or noindex pages inside the sitemap.
Can I test a robots.txt before I publish it?
Yes. Switch to Paste robots.txt, paste your draft, and test paths against it right in your browser. Nothing is sent anywhere in that mode.
Ready to grow?
Pages missing from Google?
Let's get you found.
Being found by the right people starts with a site search engines can crawl, understand and trust. Let's talk about growing your search traffic.