Robots.txt Validator — Free Online Checker

Robots.txt Validator

Robots.txt decides which pages Google, GPTBot and every other crawler may fetch. A single typo can hide your whole site — or let AI crawlers in where you did not mean to. Paste your file (or fetch it from a URL) to check the syntax, then test real URLs against the rules below.

Need to fix it first? Build a clean robots.txt with the free Robots.txt Generator · AI Crawler Checker