Robots.txt Tester
Test whether a bot may crawl a URL under your robots.txt.
- Free
- No signup
- Runs in your browser
How to use Robots.txt Tester
- Paste your robots.txt into the robots.txt box.
- Enter the URL or path you want to test, like /pricing.
- Choose a crawler, such as OAI-SearchBot or GPTBot, or pick Custom user agent to type your own.
- Read the Result panel. It says whether the crawler may crawl the path and which rule decided it.
Tip: The longest matching path wins, and Allow wins a tie, as in RFC 9309.
Result
PassOAI-SearchBot may crawl /pricing
No rule in the * group matches, so it is allowed.
Rules follow RFC 9309: the longest matching path wins, and Allow wins a tie.
Launched something?
List it on LaunchAF for a weekly launch and a permanent dofollow backlink.