Is this URL blocked, and which line did it?
Load a live robots.txt or paste your own, test any path against any crawler, and see the exact directive that decided. Then edit the file and watch the verdicts move, which is the half Google took away.
The file
Editable. Change a line and every verdict updates as you type.
Paths to test
One per line. Full URLs are fine; the query string is kept, because robots.txt matches on it.
Crawlers
The ones your file names are added automatically when you load one.
Results
8 blocked of 12*follows User-agent: */no rule matched, so allowed by default/admin/secretmatched Disallow: /admin/admin/publicmatched Allow: /admin/public/search?q=shoesmatched Disallow: /search?q=Googlebotfollows User-agent: */no rule matched, so allowed by default/admin/secretmatched Disallow: /admin/admin/publicmatched Allow: /admin/public/search?q=shoesmatched Disallow: /search?q=GPTBothas its own group User-agent: GPTBot/matched Disallow: //admin/secretmatched Disallow: //admin/publicmatched Disallow: //search?q=shoesmatched Disallow: /1 sitemap declared
- https://example.com/sitemap.xml
This tests any path against any crawler. If the question is narrower, whether the AI engines can read you at all, there is a check that answers it in one click.
Check my AI crawler accessWhy this exists
Search Console used to have a robots.txt Tester. You pasted a URL, picked a Googlebot variant, and it told you whether the URL was allowed and highlighted the line responsible. Google retired it in 2023.
What replaced it is a report, not a tester. It shows which robots.txt files Google fetched, when it last fetched them, and whether they parsed. Useful, and not the same thing: it cannot tell you whether a particular URL is allowed, and it certainly cannot tell you what would happen if you changed a line.
That second question is the one worth having a tool for. Editing a live robots.txt to find out what a rule does is a bad way to learn, because the blast radius is your whole site and the feedback arrives days later. Here the file is a textarea, the parser runs on every keystroke, and nothing you type leaves the browser.
The parser is not a model of a crawler’s behaviour. It is the one this product’s own crawler obeys when it reads someone else’s site during a site audit, which runs 42 checks and has to decide what it is allowed to fetch before it can run any of them. If it is wrong here, it is wrong there.
The three rules that catch people out
Longest match wins
Not first match, not last. Disallow: /admin with Allow: /admin/public leaves /admin/public allowed, because the Allow pattern is longer. On an exact-length tie, Allow wins.
One group applies, not all of them
A crawler reads the most specific group naming it and ignores every other, including User-agent: *. Giving a crawler a group to document a permission detaches it from your catch-all rules entirely.
Disallow is not noindex
Blocking a page stops it being fetched, not indexed. It can still appear from links elsewhere, and the crawler can no longer see the noindex you put on the page. To remove a page, allow the crawl and serve noindex.
Questions
- What happened to the Google robots.txt tester?
- Google retired the robots.txt Tester in Search Console in 2023. It was replaced by a robots.txt report that shows which files Google fetched, when, and whether they parsed. That report does not let you test a URL against your rules, which is what the old tool did and what most people are looking for when they go back to find it.
- How does robots.txt decide when two rules match?
- The most specific rule wins, measured by the length of the path pattern, and on an equal-length tie Allow beats Disallow. So Disallow: /admin and Allow: /admin/public means /admin/public is allowed, because the Allow pattern is longer. This tester implements that rule, and shows you which directive won, so you can see the reasoning rather than trusting the verdict.
- Which User-agent group applies to a crawler?
- The most specific group that matches its name, and only that group. A crawler with a group of its own reads that one and ignores User-agent: * entirely. It does not combine them. This catches people out constantly: adding a named group to document a permission silently detaches that crawler from every rule in your catch-all. The results here say which group decided, so you can tell a rule about you from the rule about everyone.
- Does robots.txt match the query string?
- Yes. Matching runs on the path and the query together, which is why Disallow: /search?q= is a real rule that works. Paste a full URL into the paths box and the query is kept.
- Do the wildcards work?
- Both of the ones crawlers support. An asterisk matches any run of characters, and a dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks PDFs and nothing else. Those are extensions to the original standard rather than part of it, but the major crawlers implement them and so does this.
- Does a Disallow remove a page from search results?
- No, and this is the most expensive misunderstanding about the file. Disallow stops a crawler fetching the page; it does not stop the URL being indexed from links elsewhere, and it prevents the crawler seeing a noindex tag you may have put on the page. To keep a page out of results, allow it to be crawled and serve noindex. Blocking is how a URL ends up listed with no description at all.
robots.txt is one check of many
The free scan reads your real pages and runs 42 of them, starting with what it is allowed to fetch.