DevKraken

Robots.txt Tester

By · Updated

Paste a robots.txt, pick a user-agent, and enter a path to see whether a crawler is allowed or blocked — using the same longest-match-wins rules Googlebot applies, with full * and $ support. Runs in your browser.

Robots.txt path tester

Blocked for *
Matched group
*
Deciding rule
Disallow: /admin/

Tip: test the same path with * and a named bot like Googlebot to see how per-crawler groups change the verdict.

How robots.txt matching works

A robots.txt is a list of groups. Each group starts with one or more User-agent lines and is followed by Allow and Disallow rules. To decide whether a crawler may fetch a URL:

  1. Pick the group. The crawler uses the group whose user-agent best matches its own name. A named match beats the catch-all *.
  2. Find the most specific rule. Every Allow/Disallow whose pattern matches the path is a candidate; the one with the longest pattern wins.
  3. Break ties with Allow. If the longest matching Allow and Disallow are the same length, the Allow wins.
  4. Default to allowed. If nothing matches, the URL is crawlable.

Build a file to test with the robots.txt generator.

Common mistakes this tester catches

  • A missing trailing slash. Disallow: /blog also blocks /blogroll and /blog.html — test both to see it.
  • Forgetting the Allow carve-out. If you block /admin/ but need /admin/public/ crawlable, the Allow must be present and the tester confirms it wins.
  • Assuming order matters. It doesn't — specificity does. A later Disallow won't override an earlier, longer Allow.
  • Wrong user-agent group. A Googlebot group makes Google ignore the * group entirely — not merge with it.

Frequently asked

How does a robots.txt tester decide allowed vs blocked?
It parses the file into user-agent groups, picks the group whose user-agent best matches the bot you entered (an exact or prefix match beats the catch-all *), then evaluates that group's Allow and Disallow rules against the path. The most specific rule — the one with the longest matching pattern — wins. If an Allow and a Disallow match with equal length, the Allow wins. If no rule matches, the path is allowed.
What do the * and $ characters mean in a rule?
An asterisk (*) matches any sequence of characters, so Disallow: /*.pdf matches any path containing .pdf. A dollar sign ($) anchors the end of the path, so Disallow: /*.pdf$ only matches paths that end in .pdf. Both are supported by Google and Bing. Without a $, a rule matches by prefix: Disallow: /admin blocks /admin, /admin/, and /administrator.
Why does Allow beat Disallow when both match?
That is Google's documented tie-break: when an Allow rule and a Disallow rule match a URL with the same number of characters, the less restrictive rule (Allow) wins. It lets you carve out exceptions — for example Disallow: /admin/ with Allow: /admin/public/ blocks the admin area but keeps the public sub-folder crawlable. This tester follows the same rule.
Does the user-agent I enter matter?
Yes. robots.txt can define different rules per crawler. The tester selects the most specific matching group: if you enter Googlebot and the file has a Googlebot group, those rules apply; otherwise the User-agent: * group is used. Try the same path with * and with a named bot to see how the verdict changes.
Can I paste a full URL instead of a path?
Yes. If you paste a full URL like https://example.com/admin/settings, the tester extracts the path and query string and tests those. robots.txt rules only apply to the path and query, not the scheme or host, so the host part is ignored.
Does a blocked path guarantee the page won't appear in Google?
No. Disallow stops crawling, but a URL that is linked from other sites can still be indexed without its content. To keep a page out of search results, allow it to be crawled and add a noindex meta tag (or X-Robots-Tag header) instead of blocking it in robots.txt.
What counts as the most specific rule?
Specificity is measured by the length of the rule's path pattern. Disallow: /admin/reports/ (16+ characters) is more specific than Disallow: /admin/ (8 characters), so for the path /admin/reports/q1 the longer rule decides the outcome. Wildcards and the end-anchor are counted as part of the pattern length.
Does this tester send my robots.txt anywhere?
No. Parsing and matching run as JavaScript in your browser. Nothing is uploaded or stored, so you can safely test rules from a production or internal site.

Related tools

All tools →