SEO Pulse Feature

Robots.txt checker
with live URL testing

Parse your robots.txt, test any URL against the rules, detect X-Robots-Tag headers — and see which crawlers are allowed or blocked for the current page.

Add to Chrome — Free All features →
What it does

Your robots.txt,
fully parsed and testable

SEO Pulse fetches your robots.txt the moment you open a page, parses every rule with syntax highlighting, and shows you a clear breakdown by user-agent. No copy-pasting into a separate tool. No switching tabs.

The URL Tester lets you enter any path and instantly see which robots.txt rules match — including which specific directive applies and why. Useful for checking whether a page you think is crawlable actually is.

X-Robots-Tag detection runs via a GET request (not HEAD — which misses headers on WordPress and CDN configurations) and surfaces any noindex or nofollow directives set at the HTTP header level.

If robots.txt returns a 403, SEO Pulse shows UNVERIFIED — not BLOCKED — and explains that Google treats a 403 as no robots.txt, meaning all URLs are crawlable.

example.com/robots.txt
# Default rules User-agent: * Disallow: /wp-admin/ Disallow: /checkout/ Allow: /wp-admin/admin-ajax.php   # AI training opt-out User-agent: GPTBot Disallow: /   User-agent: ClaudeBot Disallow: /   Sitemap: https://example.com/sitemap.xml
URL Tester — Googlebot
/products/running-shoes/ ALLOWED
/checkout/ BLOCKED
/wp-admin/admin-ajax.php ALLOWED
Features

What's included

📋
Full robots.txt parser
Fetches and parses robots.txt with live syntax highlighting. Directives colour-coded: user-agent, allow, disallow, crawl-delay, sitemap. Comments shown in muted style.
🧪
Interactive URL Tester
Enter any path and see exactly which robots.txt rules apply and why. Choose which user-agent to test against. Results update instantly.
Rule-by-rule breakdown
🤖
AI bot access grid
Visual grid showing ✓/✗ for 20 crawlers — Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bytespider, and more — for the current page URL. Each with a tooltip explaining what blocking means.
20 bots
📡
X-Robots-Tag detection
Fetched via GET request — not HEAD — to catch headers that WordPress and CDN configurations return inconsistently. Noindex and nofollow directives surfaced in the Issues Tab.
GET not HEAD
⚠️
403 handling
If robots.txt returns a 403, SEO Pulse shows UNVERIFIED — not BLOCKED. Explains that Google treats a 403 as no robots.txt, meaning all URLs are crawlable. Surfaced as a warning, not critical.
🔗
Sitemap links detected
Sitemap directives in robots.txt are extracted and linked directly. Click to open the sitemap in the Sitemap Explorer for a full recursive crawl and URL download.
FAQ

Common questions

What does a robots.txt checker do? +
A robots.txt checker fetches and parses a website's robots.txt file, shows which crawlers are allowed or blocked, and lets you test specific URLs to see which rules apply. SEO Pulse does this automatically whenever you open a page, with no copy-pasting or extra steps.
What is the X-Robots-Tag and how is it different from robots.txt? +
Robots.txt controls which pages crawlers can access. The X-Robots-Tag is an HTTP response header that controls indexing at the individual URL level. A page can be fully crawlable via robots.txt but still have a noindex X-Robots-Tag — meaning it gets crawled but not indexed. SEO Pulse detects both, and fetches X-Robots-Tag via a GET request rather than HEAD to avoid missing headers on WordPress and CDN servers.
What happens if robots.txt returns a 403 error? +
Google treats a 403 on robots.txt as if no robots.txt exists — meaning all URLs are crawlable. SEO Pulse shows UNVERIFIED status (not BLOCKED) and surfaces a warning in the Issues Tab explaining this behaviour. This distinction matters: showing BLOCKED would suggest pages are protected when they may not be.
Can I test whether a specific URL is blocked? +
Yes. The URL Tester in the Robots tab lets you enter any path and see which rules apply, which specific directive matched, and whether the URL is allowed or blocked — for any user-agent in your robots.txt.
Does it check AI crawlers like GPTBot and ClaudeBot? +
Yes. The AI bot access grid checks robots.txt rules for 20 crawlers including GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, CCBot, Bytespider, Google-Extended, Applebot-Extended, and more. Each bot shows ✓ allowed or ✗ blocked for the current page URL, with a tooltip explaining what that bot does and what blocking it means for your site. Learn more about the AI bot access feature →
Related features

More in SEO Pulse

🛡️
AI Bot Access
Visual grid for 20 crawlers with tooltips
🚨
Issues Tab
85+ checks ranked by severity
📋
On-Page SEO Checker
Title, meta, canonical, headings and more
🔄
Redirect Checker
Real-time chain tracker with status codes

Check robots.txt on any page instantly

Install SEO Pulse, open a page, open the Robots tab. No setup.

Add to Chrome — Free All features →

★★★★★ on Chrome Web Store · 100+ users · No data sent to servers